MyPy third-party requirements support is broken for Python 2

[Eric-Arellano]

Right now, we expect for the interpreter constraints to be different for the tool_pex and runner_pex than they are for the requirements_pex. This is because MyPy must run with Python 3.5, but the user's code may be anything, including Py27.

We use --pex-path to join all the PEXes. However, this results in error messages like this:

Failed to execute PEX file. Needed macosx_10_15_x86_64-cp-36-cp36m compatible dependencies for:
 2: filelock
    But this pex only contains:
      filelock-3.0.12-py2-none-any.whl

It appears that whatever interpreter is used to run the runner PEX, it expects for every requirement to be compatible with it. So, running MyPy with Py36 complains when it can only find Py27 wheels.

$ pex --interpreter-constraint='CPython==2.7.*' -o requirements.pex filelock
$ pex --interpreter-constraint='CPython==3.6.*' -o runner.pex --pex-path=requirements.pex
$ ./runner.pex
Failed to execute PEX file. Needed macosx_10_15_x86_64-cp-36-cp36m compatible dependencies for:
 1: filelock
    But this pex only contains:
      filelock-3.0.12-py2-none-any.whl

Instead of using --pex-path to join the requirements.pex, we need a mechanism to get the distributions out of the PEX and treat them like normal data. Then, in our MyPy code, we can load up those dists.

---

Alternatively, we can set PEX_IGNORE_ERRORS...this fixes the issue, but I'm not sure if it's safe to do.

[Eric-Arellano]

Put up a failing test at #10820. PEX_IGNORE_ERRORS does not work. While the runner Pex no longer errors, it cannot load the Python 2-only wheel, which results in MyPy complaining about missing site packages, even tho the library does have type hints.

Taking a step back..that makes sense. How would MyPy—which uses >=3.5—load a Python 2-only wheel? It can parse Python 2 code using the typed-ast library, but this doesn't necessarily mean it can handle Python 2-only distributions?

I think that this could suggest one of two things:

1. If you are using Py2, you'll only have type stubs if the wheel is also compatible with Py3. Otherwise, you'll have to ignore that dependency.
2. If you're running with Py2, perhaps we should include Py3 in the interpreter constraints when resolving requirements. That way, we could have both the Py2 and Py3 wheels for the library.
   • Although, this sounds like a recipe for disaster. For example, if you have a dist that cannot be installed with Py3, now the build is broken.

#1 might be adequate, but it does raise some concerns I have when we're purely talking about Python 3. Imagine you resolve your requirements with Py36, but run MyPy with Py35. Now, none of your requirements will load. Yes, we can use PEX_IGNORE_ERRORS, but you'll get a bunch of MyPy failures about missing requirements.

--

Instead, what I think this is suggesting is that for Py3-only, the tool Pex and requirement Pex need to have the same constraints. Perhaps we need to use something like we do with iPython to determine the constraints:

pants/src/python/pants/backend/python/goals/repl.py

Lines 56 to 64 in dadbcc5

	# Note that we get an intermediate PexRequest here (instead of going straight to a Pex)
	# so that we can get the interpreter constraints for use in ipython_request.
	requirements_pex_request = await Get(
	PexRequest,
	PexFromTargetsRequest,
	PexFromTargetsRequest.for_requirements(
	(tgt.address for tgt in repl.targets), internal_only=True
	),
	)

Iff we're using Python 2, we set PEX_IGNORE_ERRORS to workaround some of the dists being missing.

[cosmicexplorer]

How would MyPy—which uses >=3.5—load a Python 2-only wheel? It can parse Python 2 code using the typed-ast library, but this doesn't necessarily mean it can handle Python 2-only distributions?

So this makes sense, but I feel like if this was true, it would make it really difficult for people to use mypy on py2-only projects in general! I know pip was py2-compatible for a while, but it also vendored everything it needed. I'm going to see if I can find an example of a project that:

1. is py2-compatible
2. uses mypy
3. pulls in 3rdparty dists

[cosmicexplorer]

Ok, so after continuing your awesome work at pex-tool/pex#1032, I think this line was more insightful than I realized at first:

Instead of using --pex-path to join the requirements.pex, we need a mechanism to get the distributions out of the PEX and treat them like normal data. Then, in our MyPy code, we can load up those dists.

Could we perhaps just unconditionally unzip the requirements pex? And in fact, even better: instead of adding complex round-trip logic to extract requirements out of a pex file after the PEXBuilder has already done all the hard work to pack the distributions into the zip, could we instead invoke the pex library API!

We could make this a separate rule which uses a script that works in isolation, without being able to import anything else from pants -- just like the fabled ipex launcher at https://github.com/pantsbuild/pants/blob/1.25.x-twtr/src/python/pants/backend/python/subsystems/ipex/ipex_launcher.py. That script could call pex.resolver.resolve() to download distributions, but instead of creating a pex file, literally just stop there. This should make the process completely cacheable, and the output digest transferable to the downstream mypy invocation.

This would be much faster (I think) than creating a whole pex file from the dependencies, in addition to avoiding complex work to extract anything out of the pex format. I'll look to extend #10820 now or tomorrow with this idea.

[cosmicexplorer]

It looks like we might be able to remove this lengthy comment and the monkey-patching here too!

pants/src/python/pants/backend/python/typecheck/mypy/rules.py

Lines 71 to 73 in 18c0a56

	# its search to `site-packages`. Since PEX deliberately isolates itself from `site-packages` as
	# part of its raison d'être, we monkey-patch `site.getsitepackages` to look inside the scrubbed
	# PEX sys.path before handing off to `mypy`.

[cosmicexplorer]

Hmmmm. Before I get too much further into this, I think this might be a duplicate of #9206!

[cosmicexplorer]

I think it's a little different than #9206, although that would be half the battle. In that case, we just needed to shade requirements from tools. In this case, I'm not sure (like @Eric-Arellano mentioned above) whether mypy is able to load python 2 dists even if they're on the sys.path (because that sounds like it would be fraught with issues). We should figure out whether mypy can load python 2 wheels for typing info, or if it can find it if we expose it on sys.path (which would be closer to what we do right now).

However, I want to note that if we can make the current method work, we should be able to generalize it into a solution for #9206, which would be cool.