Split Python targets into atom vs. generator and use source: str field

[Eric-Arellano]

Now our Python code operates on python_source and python_test targets. This causes some changes.

Dependency inference

Dependency inference no longer runs on python_sources and python_tests, only python_source and python_test. That applies to import analysis, conftest.py, and __init__.py.

This actually speeds up dependency inference! We no longer run import_parser.py twice on the same files.

Before:

❯ hyperfine -r 10 './pants --no-process-execution-local-cache --no-pantsd dependencies src/python::'
Benchmark #1: ./pants --no-process-execution-local-cache --no-pantsd dependencies src/python::
  Time (mean ± σ):      7.650 s ±  0.741 s    [User: 5.794 s, System: 2.141 s]
  Range (min … max):    7.250 s …  9.673 s    10 runs

After:

❯ hyperfine -r 10 './pants --no-process-execution-local-cache --no-pantsd dependencies src/python::'
Benchmark #1: ./pants --no-process-execution-local-cache --no-pantsd dependencies src/python::
  Time (mean ± σ):      5.998 s ±  0.279 s    [User: 3.524 s, System: 0.903 s]
  Range (min … max):    5.708 s …  6.688 s    10 runs

It means that ./pants dependencies src/py:lib now only has explicitly declared dependencies. You have to use --transitive to get the results of dep inference.

Thanks to this change, we can simplify import_parser.py to assume there is only one file.

Lockfile generation

When determining the interpreter constraints to use, we now look at generated targets, not target generators. We only run over the python_source and python_test targets that actually will be used.

This makes us future-proof to an overrides field adding skip_tool to only some of the files. I think it also fixes our dependency resolution for Pylint looking at direct deps?

FieldSet.opt_out no longer worries about file targets

Pytest and MyPy used to skip running on non-file targets. That can be removed now. This unblocks allowing you to explicitly define a python_source/python_test target.

[Eric-Arellano]

Each commit is distinct.

--

I'm still thinking of a good title to put in the changelog to capture the change to dependency inference. Maybe

python_sources and python_tests targets no longer use dependency inference, only python_source and python_test targets

Fixing https://docs.google.com/document/d/1WWQM-X6kHoSCKwItqf61NiKFWNSlpnTC5QNu3ul9RDk/edit# will be important I think, as ./pants dependencies :: no longer includes the generated targets given that we don't "expand" the input roots

[Eric-Arellano]

Also, the dependency inference speedup is an excellent example of why I felt so compelled to prioritize this change. As the author of that dep inference implementation, I had no idea we were running the import parser twice! Probably I could have profiled better, but also it's hard to reason about python_library targets generating python_library targets. I think the modeling is much more intuitive now.

[stuhood]

Thanks!

This actually speeds up dependency inference! We no longer run import_parser.py twice on the same files.

Whoops.

Fixing https://docs.google.com/document/d/1WWQM-X6kHoSCKwItqf61NiKFWNSlpnTC5QNu3ul9RDk/edit# will be important I think, as ./pants dependencies :: no longer includes the generated targets given that we don't "expand" the input roots

Yea, fixing Specs matching is a blocker for 2.8.0.

[Eric-Arellano]

Confirmed that depending on a python_sources target still results in it being an alias for all of its generated targets. Which means that we do not break Pylint, which solely looks at direct deps and not transitive. Great! I was afraid I broke that.