Provide way to infer deps from environment-provided dependencies

[jriddy]

Is your feature request related to a problem? Please describe.
The Pants Python backend doesn't currently only provides ways to deal with dependencies that come from first-party in-repo code, or third-party code via the Pip/PyPI ecosystem. However there are many valid ways of running and using Python code that involve dependencies being provided by systems outside this ecosystem.

Some (non-hypotheitical) examples include:

• Dependencies provided by system packages such as DEB or RPM packages
• Dependencies provided by a deployment runtime, such as implicit Lambda layers as seen in Allow Lambda targets to point to python_aws_lambda_layers and exclude code automatically #19256
• Dependencies provided via Conda

These are all actual known use cases. It's also easy to imagine that proprietary on nonstandard code might be distributed via ad-hoc mechanisms as well, but I don't know of any specific outstanding requests for dealing with this (although I have seen this in the past with vendor-provided drivers and bindings).

Right now, there's simply no way to reference these kinds of dependencies within Pants. Existing workarounds are to either manually map an environments dependencies to (close) Pip-based equivalents, to use no-infer-dep pragmas to hide these deps, or to simply lie to Pants and create false dependencies that represent code we can't actually resolve, and just avoid trying to package these deps via some kinda transitive exclude. All of these are hacky and annoying.

Describe the solution you'd like
We could provide a union rule for Python dep inference that allows us to model other kinds of third-party dependencies (fourth party dependencies? 😁). I suggested this in a Slack thread, and it didn't receive any major objection.

The current module mapping mechanism hard codes the two paths but this could be extended to be a union membership, similar to the Go implementation or even the Python first party implementation.

The goal here would be to leave this open ended to let other rules provide their own implementation of a module->target mapping as they see fit, and in so doing, reify those external dependencies for Pants in some way.

Describe alternatives you've considered
The main alternatives are listed above: you have to hack around Pants to do this. This is okay for some one-off cases, but it quickly gets unwieldy.

Additional context
Deferring dependencies to the environment almost necessarily breaks Pants' guarantees around hermeticism and reproducibility. I'm not proposing any particular way to deal with this for this issue, because it really depends on the provider of the dependency how this could be accomplished. This probably needs to be documented clearly somehow for consumers of this union rule, so any accompanying docs should state this clearly.

There's also an issue of interpreter constraints. System provided dependencies are unlikely to apply to more than one interpreter, and may only make sense when using a particular interpreter, even of the same version, since site configurations will vary between interpreters and thus the nature of the builtin module search path and import hooks. I'm not sure how much interpreter selection should fall into this issue, but it's certainly a complicating factor, and we should probably at least research how this will play out.

Lastly, it may not be possible to use any of these dependencies with any of the default implementations of Python goals without making PEX more capable of excluding dependencies in a fine grained way as described in pex-tool/pex#2097

[huonw]

With pex 2.1.151 (https://github.com/pantsbuild/pex/releases/tag/v2.1.151), which Pants now includes (#20149), there's a flag --exclude that could be used as a building block here: if Pants knows which dependencies are provided by a target environment, they can be --exclude'd. That said, we may prefer to wait for --provided (pex-tool/pex#2097 (comment)) too, so that Pants can (optionally) have PEX actually verify the deps exist.