Relax check_dist_restriction for dry run

[tgeng]

There does not seem to be any reason to require --only-binary or --no-deps in dry run mode. There is also no need to specify --target in such a situation.

With the new --report flag, this makes it possible to use pip just as a cross-platform dependency resolver.

Context: #11764

[sbidoul]

There does not seem to be any reason to require --only-binary

Is it not necessary to ensure pip will not attempt a metadata preparation step for some sdist, which it can only do on the target platform?

[tgeng]

There does not seem to be any reason to require --only-binary

Is it not necessary to ensure pip will not attempt a metadata preparation step for some sdist, which it can only do on the target platform?

Thanks! Actually I am not familiar with how dependency resolution works. Does this mean in theory dependency resolution can invoke arbitrary native code? Is there an example that has such requirement do that I can dig into with? Maybe we can let it fail in that case if such situations are relatively rare? That way at least dry run can somewhat work cross platform.

[pfmoore]

Yes, easily. Any dependency that has no wheel, which is built using setuptools, will invoke setup.py when being built. The resolution process needs dependency data, which requires a build, because it is not available statically.

[pfmoore]

And the implication of that is that it’s extremely common, so failing is not an option.

[sbidoul]

When PEP 643 compliant sdists will be more common, there will be the possibility to inspect dependencies in the sdist metadata, when they are declared static (which should probably be the most common case).

And if there is a source tree in the dependencies, pip would need to grow a way to inspect pyproject.toml to see if the dependencies and version are static, and get them from there.

Also you are going to hit #11664 and friends.

So yeah, there is a path to cross-platform dependency resolution, but we are not quite there yet.

BTW, @pfmoore I'm wondering if there is a way to measure PEP 643 adoption on PyPI?

[pfmoore]

BTW, @pfmoore I'm wondering if there is a way to measure PEP 643 adoption on PyPI?

Easy, at the moment it's zero, because the issue for supporting Metadata 2.2 hasn't been implemented yet 🙁

Once that's been dealt with, build backends need to start generating Metadata 2.2. For all but setuptools, that will be easy, and setuptools is getting there (with static config support) but it'll be a while I suspect. Then you'll be able to download and check all sdists on PyPI to get a feel for the answer 🤣 When PEP 658 gets implemented for sdists (there's this issue to cover doing it for wheels, but nothing yet for sdists) the information will be available from the simple API. But even then, it will be a long time before we can assume that a resolve will never need to look at a pre-metadata 2.2 sdist to complete (existing sdists on PyPI will not be retroactively updated, obviously).

All of the parts are there, but they need implementing. And there's still legacy.

So basically we won't be able to avoid the possibility of running build code as part of a resolve for the foreseeable future. What we will be able to do is avoid running code or downloading full distributions in the common cases, which will be a significant performance win for pip.

Edit: To give a less facetious answer, PEP 643 isn't supported yet on PyPI, so all existing sdists need a build step, and will forever (as changing uploaded artifacts isn't allowed). Once PEP 643 support is added, we'll hopefully see the majority of new sdists with static metadata for everything the resolver needs, so the adoption rate will be essentially new vs old. I don't know when we'll reach a tipping point where we can declare all non-static cases "sufficiently old" that we're happy for the resolver to error if it hits one.

[tgeng]

I see. Thanks all for the explanation! Please feel free to close the issue and PR as there seems to be no feasible way to do cross platform resolution correctly for the foreseeable future.