Use packaging.tags for doing compatible wheel tag calculation #7354

chrahunt · 2019-11-15T02:06:32Z

Introduction

packaging.tags provides a simple sys_tags function for getting applicable tags for the running interpreter, but it does not allow customization of arguments. Since pip allows users to provide explicit interpreter (e.g. "pp"), platform (e.g. "linux_x86_64"), abi (e.g. "abi3"), and version (e.g. "35"), we have to use a different interface.

packaging.tags provides three functions for users that need to customize tag generation:

cpython_tags - for tags applicable to CPython only
generic_tags - for tags applicable to any non-CPython Python implementation
compatible_tags - for tags that are not specific to an interpreter
implementation

cpython_tags and generic_tags are mutually exclusive, and return tags that are the highest priority. These tags are the most specific.

compatible_tags are always applicable, and a lower priority since they may just be compatible with e.g. the Python language version, but lack optimizations available in compiled code available for a specific interpreter/ABI/platform.

In this PR we remove most of our custom logic in pip._internal.pep425tags, instead relying on these three functions from packaging.tags.

Approach

It's a given that we want to switch to packaging.tags for calculating compatible wheel tags, but while doing that we have a few other goals:

minimize bugs merged by making changes easy to review
if possible, compare the implementation in packaging.tags with what is currently in pip so we can understand possible issues that may be raised later

To satisfy 1, the PR is broken into small commits that should each be easy to recognize the literal changes of and confirm as equivalent or understand the scope of change. Each commit should pass lint checks and tests, which should reduce the number of trivial things that need to be worried about and ensure that all changes are self-contained.

To satisfy 2, we take an approach that transforms pep425tags into something that looks closer to packaging.tags before actually using the functions.

For each of the 3 custom tags functions, we make a copy of get_supported, then:

Refactor it locally, taking into account its new, more limited, responsibilities
Introduce the packaging.tags function for the case where there are no custom arguments provided
Customize arguments one-by-one and delegate to the packaging.tags function
When there is no pip-specific logic left, remove the intermediate function and use the packaging.tags function directly in get_supported

Following that, we remove unused utility methods, then in the end we're left with an implementation that relies solely on the tag generation in packaging.tags.

Fixes #6908.

pradyunsg · 2019-11-15T03:06:15Z

You can probably do the following, to make the vendoring work properly.

diff --git a/tools/automation/vendoring/__init__.py b/tools/automation/vendoring/__init__.py
index 1c2ae3a6..a77e9e58 100644
--- a/tools/automation/vendoring/__init__.py
+++ b/tools/automation/vendoring/__init__.py
@@ -128,6 +128,7 @@ def vendor(ctx, vendor_dir):
     )
     remove_all(vendor_dir.glob('*.dist-info'))
     remove_all(vendor_dir.glob('*.egg-info'))
+    remove_all(vendor_dir.glob('*/py.typed'))
 
     # Cleanup setuptools unneeded parts
     (vendor_dir / 'easy_install.py').unlink()

BrownTruck · 2019-11-17T02:45:02Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

tools/automation/vendoring/__init__.py

chrahunt · 2019-11-23T23:09:46Z

Test failure would be fixed by #7395.

src/pip/_internal/pep425tags.py

chrahunt · 2019-11-24T15:55:12Z

Updated for latest commit, but had to update vendoring script locally to work around pypa/flit#298.

BrownTruck · 2019-11-26T05:00:03Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

chrahunt · 2020-01-05T17:47:41Z

Rebased on master and updated to use the latest commit on master for packaging.

When packaging is released, I can start breaking pieces of this off into separate PRs.

Since this function is only called for non-CPython interpreters, the abi3-related branches are not relevant.

As with cpython_tags and compatible_tags, we assume this function covers our use cases in the no-argument case and will customize our arguments for it in a few steps.

Now that we're fully using packaging.tags, everything in the supported list is already Tag. Further, since each function is responsible for its own set of non-overlapping tags, we can also remove the de-duplication.

Since we only call this function when platform is not None, we can drop one of the branches. This is the only place using our manylinux auto-deduction functions, so those can be removed next.

The remaining glibc-related functions are required to allow us to put the libc version in the pip user agent (for stats).

Previously, these were used when the user did not provide an explicit ABI. Now this is handled internally in `packaging.tags` (by `_cpython_abi` and `_generic_abi`).

Now handled internally in `packaging.tags` (in `_platform_tags`).

pradyunsg · 2020-01-08T14:21:50Z

src/pip/_internal/pep425tags.py

-    # type: (str) -> Optional[str]
-    return sysconfig.get_config_var(var)
-
-
 def version_info_to_nodot(version_info):


follow up: I suggest moving the call to be changed -- changing get_supported to allow for version to be None (and use this value in that case).

xavfernandez

Great job 👍

pradyunsg · 2020-01-09T04:40:40Z

Indeed. This was a decently large chunk of work which required a fair bit of figuring out the right way, and I know that you've spent time and effort to make the PRs easy to review!

Thanks @chrahunt for driving this and making it happen!

chrahunt added the type: maintenance Related to Development and Maintenance Processes label Nov 15, 2019

chrahunt changed the title ~~WIP: Use packaging.tags~~ WIP: Replace pep425tags with packaging.tags Nov 15, 2019

chrahunt force-pushed the maint/use-packaging-tags branch from 6878a93 to 8de85bb Compare November 16, 2019 19:19

chrahunt mentioned this pull request Nov 17, 2019

Factor platform getter functions out of get_supported #7370

Merged

BrownTruck added the needs rebase or merge PR has conflicts with current master label Nov 17, 2019

chrahunt force-pushed the maint/use-packaging-tags branch from 8de85bb to cf52515 Compare November 19, 2019 02:44

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Nov 19, 2019

pradyunsg reviewed Nov 19, 2019

View reviewed changes

tools/automation/vendoring/__init__.py Outdated Show resolved Hide resolved

chrahunt force-pushed the maint/use-packaging-tags branch 2 times, most recently from d57eb4a to 72432af Compare November 23, 2019 23:03

chrahunt force-pushed the maint/use-packaging-tags branch from 72432af to 3601c28 Compare November 23, 2019 23:37

chrahunt commented Nov 24, 2019

View reviewed changes

src/pip/_internal/pep425tags.py Outdated Show resolved Hide resolved

chrahunt mentioned this pull request Nov 24, 2019

pip download should handle ABI3 tags better #6070

Closed

chrahunt force-pushed the maint/use-packaging-tags branch from 3601c28 to 6731511 Compare November 24, 2019 15:54

takluyver mentioned this pull request Nov 24, 2019

flit_core uses pip-unsupported backend-path pypa/flit#298

Closed

BrownTruck added the needs rebase or merge PR has conflicts with current master label Nov 26, 2019

xavfernandez mentioned this pull request Jan 2, 2020

Expose various function in tags publicly pypa/packaging#231

Merged

chrahunt force-pushed the maint/use-packaging-tags branch from 6731511 to 56ee682 Compare January 5, 2020 17:43

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Jan 5, 2020

chrahunt force-pushed the maint/use-packaging-tags branch from 56ee682 to 13c6934 Compare January 6, 2020 22:04

chrahunt added 20 commits January 7, 2020 18:59

Customize python_version for packaging.tags.cpython_tags

147680a

Customize abis for packaging.tags.cpython_tags

05045e7

Customize platforms for packaging.tags.cpython_tags

fecfadb

Make packaging.tags.cpython_tags unconditional

56840c3

Inline packaging.tags.cpython_tags

1574872

Remove unused abi3 branch in _generic_tags

fa1ec40

Since this function is only called for non-CPython interpreters, the abi3-related branches are not relevant.

Use packaging.tags.generic_tags

281273d

As with cpython_tags and compatible_tags, we assume this function covers our use cases in the no-argument case and will customize our arguments for it in a few steps.

Customize interpreter for packaging.tags.generic_tags

77dbd27

Customize abis for packaging.tags.generic_tags

0bebeb6

Customize platforms for packaging.tags.generic_tags

293b778

Make packaging.tags.generic_tags unconditional

72dcd34

Inline packaging.tags.generic_tags

3e66ab0

Remove unnecessary conversion in get_supported

ad546b5

Now that we're fully using packaging.tags, everything in the supported list is already Tag. Further, since each function is responsible for its own set of non-overlapping tags, we can also remove the de-duplication.

Simplify _get_custom_platforms

9b34435

Since we only call this function when platform is not None, we can drop one of the branches. This is the only place using our manylinux auto-deduction functions, so those can be removed next.

Remove unused manylinux auto-deduction functions

2455977

Remove unused glibc functions

7aaa705

The remaining glibc-related functions are required to allow us to put the libc version in the pip user agent (for stats).

Remove unused abi functions

896317d

Previously, these were used when the user did not provide an explicit ABI. Now this is handled internally in `packaging.tags` (by `_cpython_abi` and `_generic_abi`).

Remove unused get_platform function

2b1b60f

Now handled internally in `packaging.tags` (in `_platform_tags`).

Remove unused version functions

ae21af7

Add news

d7fda71

chrahunt force-pushed the maint/use-packaging-tags branch from f146ee0 to d7fda71 Compare January 8, 2020 00:01

pradyunsg approved these changes Jan 8, 2020

View reviewed changes

xavfernandez approved these changes Jan 8, 2020

View reviewed changes

xavfernandez merged commit fe3e892 into pypa:master Jan 8, 2020

chrahunt deleted the maint/use-packaging-tags branch January 9, 2020 04:54

This was referenced Jan 21, 2020

Pip 20.0.0+ compatibility pygfx/wgpu-py#46

Closed

Cannot install platform-specific wheels with non-python extension binaries on pip>=20.0.0 #7636

Closed

lock bot added the auto-locked Outdated issues that have been locked by automation label Feb 8, 2020

lock bot locked as resolved and limited conversation to collaborators Feb 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use packaging.tags for doing compatible wheel tag calculation #7354

Use packaging.tags for doing compatible wheel tag calculation #7354

chrahunt commented Nov 15, 2019 •

edited

Loading

pradyunsg commented Nov 15, 2019

BrownTruck commented Nov 17, 2019

chrahunt commented Nov 23, 2019

chrahunt commented Nov 24, 2019

BrownTruck commented Nov 26, 2019

chrahunt commented Jan 5, 2020

pradyunsg Jan 8, 2020

xavfernandez left a comment

pradyunsg commented Jan 9, 2020

Use packaging.tags for doing compatible wheel tag calculation #7354

Use packaging.tags for doing compatible wheel tag calculation #7354

Conversation

chrahunt commented Nov 15, 2019 • edited Loading

Introduction

Approach

pradyunsg commented Nov 15, 2019

BrownTruck commented Nov 17, 2019

chrahunt commented Nov 23, 2019

chrahunt commented Nov 24, 2019

BrownTruck commented Nov 26, 2019

chrahunt commented Jan 5, 2020

pradyunsg Jan 8, 2020

Choose a reason for hiding this comment

xavfernandez left a comment

Choose a reason for hiding this comment

pradyunsg commented Jan 9, 2020

chrahunt commented Nov 15, 2019 •

edited

Loading