Add "generic" IDE support via virtualenv and codegen export

[stuhood]

(relates to #11151 and #10920)

[stuhood]

https://www.pantsbuild.org/docs/setting-up-an-ide describes the current state of the world.

[stuhood]

@benjyw will take a swing at this.

[joshua-cannon-techlabs]

The Issue notes don't specify an implementation, so I'll throw in my 2 cents. (For completeness, knowing I use VS Code but I'm not sure my 2 cents is IDE-specific).

Our org has >200 requirements and some are huge, so just from a sanity standpoint just making and maintaining a virtual environment with the reqs installed is not a solution.

This also falls short because I believe some debug tools won't like that I set a breakpoint in a library in the virtual environment, but the real implementation running in pants is under some other path.

So ideally, the virtual environment would just be a cleverly empty one with PYTHONPATH or pyvenv.cfg or some other "go look over here" mechanism. That would reduce the overhead and ensure there's only one true path.

I know the requirements are in a PEX file somewhere, but since that's just a zip I would hope 🙏 Python is OK with that (after all it's ok executing from a ZIP, so why not a zipped site-packages?)

---

Last thing, and this is more of a "really-nice-to-have" (but it would be really nice to have 😄) have this be maintained behind a setting I can enable which implies this shim little virtual environment is updated when requirements.pex is being managed (in the case that there would be work to be done on changing requirements.txt). Then no one has to remember to update the environment when requirements.txt changes. pants "just handles" it for me <3

[benjyw]

Thanks for the notes, @joshua-cannon-techlabs !

Some of what you're asking for might require an IDE plugin (which is planned, but a few months away). The idea for this ticket was a quick-and-dirty mechanism to export useful information that you manually point your IDE at.

Also, I know PyCharm well but not VS Code, so some of my thinking might be biased by that.

The plan was to create a virtualenv with all the requirements in the repo (or all the requirements for the targets you care about), and point the IDE at that and say "index this". IIUC you're saying that you have so many requirements that creating that venv would take a prohibitive amount of time? Presumably though Pants has already done that resolve at least once?

If you have a constraints.txt for example, we could reuse the "repository pex" we create from that. But we still need to materialize it somewhere on disk. It doesn't exist at some well-known disk location until you do so, it just lives as a ghost in the cache...

[joshua-cannon-techlabs]

The plan was to create a virtualenv with all the requirements in the repo (or all the requirements for the targets you care about), and point the IDE at that and say "index this". IIUC you're saying that you have so many requirements that creating that venv would take a prohibitive amount of time? Presumably though Pants has already done that resolve at least once?

Precisely! With the missing link here being the virtualenv would just be a "pointer" to pants' cached version of the dependencies. Ideally I download things 0 times but that's impossible, so next best thing would be to only download them once. And since pants needs to download them, I want it to be the sole provider of dependencies.

I do realize now what I'm suggesting only works if you have a constraints.txt, or have done something similar to tell pants about every conceivable dependency. But, again, if the user wants a virtualenv that works with all their dependencies they need pants to manage that anyways.

---

So I guess a rough sketch of this would be:

• If: the user has constraints.txt set:
  • Resolve requirements.pex from it if needed (or pull from the cache)
• Otherwise: If the user doesn't have constraints.txt set:
  • Resolve requirements.pex to be the transitive set of all third-party dependencies (or cached?)
• Create/update the virtualenv with a "pointer" to requirements.pex

[Eric-Arellano]

I do realize now what I'm suggesting only works if you have a constraints.txt, or have done something similar to tell pants about every conceivable dependency. But, again, if the user wants a virtualenv that works with all their dependencies they need pants to manage that anyways.

Something I continue to be confused with is how IDE integration will work when we support multiple user lockfiles. Iiuc, IDEs want a single virtualenv, but the idea of multiple user lockfiles is that you may have multiple disjoint lockfiles that aren't compatible with each other.

[stuhood]

I do realize now what I'm suggesting only works if you have a constraints.txt, or have done something similar to tell pants about every conceivable dependency. But, again, if the user wants a virtualenv that works with all their dependencies they need pants to manage that anyways.

Something I continue to be confused with is how IDE integration will work when we support multiple user lockfiles. Iiuc, IDEs want a single virtualenv, but the idea of multiple user lockfiles is that you may have multiple disjoint lockfiles that aren't compatible with each other.

Most likely we should fail if you try to open an incompatible subset in one IDE project. Essentially, if you run ./pants export-strawname-venv has-one-resolve:: has-another-resolve::, we should fail indicating that they don't share a compatible resolve. Because AFAIK, IDEs don't support this case either: you'd need to switch "projects" to allow a different set of thirdparty deps.

[benjyw]

See #13415 for a first pass. @joshua-cannon-techlabs, if you have a moment could you test it out on your repo and see if it's viable (let me know if you need info on how to run Pants from sources)? It basically symlinks to a virtualenv that is pretty likely to already exist if you have resolved your repo's requirements at any point (e.g., if you have a constraints file set up).

[joshua-cannon-techlabs]

By the end of this, I'll have collected all y'all's forks like infinity stones:

(ve) joshuacannon@CEPHANDRIUS:~/work/pants$ git remote
benjyw
origin
stuhood

[joshua-cannon-techlabs]

LGTM

[stuhood]

@benjyw : Can we call this resolved by #13415?

[benjyw]

Still need to add codegen export into the export goal, let's keep this open as a reminder of that.