Support using pip-compile generated file header to choose Python version #4216
Comments
As far as I know `pip-compile` only needs to be run with the right version of Python, it doesn't actually need to be run in the same virtualenv that the requirements file is (or will be) installed in. IIRC the only reason why I did use `tox` to run `pip-compile` in the first place was that we might have a project that uses different Python versions for different virtualenvs. For example we did this in the past with legacy Via: Via itself only supported Python 2 so virtualenvs like dev, tests, functests, etc used Python 2. But the Black code formatter could only be run in Python 3 (even though it could format Python 2 code) so we used Python 3 for the format virtualenv. Generally speaking the ability to have different tox environments using different Python versions enables us to adopt a tool like formatter or linter without that tooling necessarily having to be run with the same version of Python that the app runs in. But: 1. We don't currently have any apps that use multiple Python versions in this way. 2. As far as I know using multiple Python versions with `pip-compile`'d requirements files isn't compatible with Dependabot: Dependabot will use the first Python version from your `.python-version` to compile *all* your `requirements.txt` so it'll break any `requirements.txt` files that use a different Python version. I don't understand the Dependabot Ruby source code well enough to verify whether this is still the case but last time I tested it was and there are still multiple open issues about this on Dependabot's GitHub issues, for example dependabot/dependabot-core#4216 and dependabot/dependabot-core#4607 I'm not sure how we got away with this when legacy Via used multiple Python versions. I suspect maybe it wasn't using Dependabot? I think we should just consider leaning in to the fact that while our Python _packages_ can support multiple Python versions because they don't use `pip-compile` and don't have pinned requirements files, our _apps_ can only support one version of Python at a time (one version shared by all their tox envs). If we're willing to make that assumption we can simplify the implementation of `make requirements`, which is what this commit does.
`pip-compile` only needs to be run with the right version of Python, it doesn't actually need to be run in the same virtualenv that the requirements file is (or will be) installed in. IIRC the only reason why I did use `tox` to run `pip-compile` in the first place was that we might have a project that uses different Python versions for different virtualenvs. For example we did this in the past with legacy Via: Via itself only supported Python 2 so virtualenvs like dev, tests, functests, etc used Python 2. But the Black code formatter could only be run in Python 3 (even though it could format Python 2 code) so we used Python 3 for the format virtualenv. Generally speaking the ability to have different tox environments using different Python versions enables us to adopt a tool like a formatter or linter without that tool necessarily having to be run with the same version of Python that the app runs in. But: 1. We don't currently have any apps that use multiple Python versions in this way. 2. As far as I know using multiple Python versions with `pip-compile`'d requirements files isn't compatible with Dependabot: Dependabot will use the first Python version from your `.python-version` to compile *all* your `requirements.txt` files so it'll break any `requirements.txt` files that use a different Python version. I don't understand the Dependabot Ruby source code well enough to verify whether this is still the case without actually trying it but [last time I tested](https://github.com/seanh/python-multiversion-dependency-management-demo#dependabot-) it was and there are still multiple open issues about this on Dependabot's GitHub issues, for example dependabot/dependabot-core#4216 and dependabot/dependabot-core#4607 I think it may be possible to get Dependabot to use the right Python version if you put different requirements files in different subdirs and give each subdir its own `.python-version` file. But I haven't tested this and I don't think we want to do it. I'm not sure how we got away with this when legacy Via used multiple Python versions. I suspect maybe it wasn't using Dependabot? I think we should consider leaning in to the fact that while our Python _packages_ can support multiple Python versions because they don't use `pip-compile` and don't have pinned requirements files, our _apps_ can only support one version of Python at a time (one version shared by all their tox envs). If we're willing to make that assumption we can simplify the implementation of `make requirements`, which is what this commit does.
|
👋 Interesting idea. What do you think about moving more toward extracting the That seems to be the way a lot of tooling in the python ecosystem is moving, so is a solution that's more generic than |
Yeah I think that makes sense. I'm not sure there is a standard way to define python version in pyproject.toml though. It might be OK to take the lower bound of |
That wouldn't solve the use case where there are multiple lockfiles each targeting a different Python version (see #4607). |
deivid-rodriguez
added
L:python:pip-compile
|
I think parsing the python version from the header and using that as proposed would be the most reliable way to update |
|
The Python version is just one small part of the problem. How about the platform/arch factors? #4607 |
|
@deivid-rodriguez thanks for implementing this. Do you know how to tell whether it's deployed to Github yet? I just tried it on a Github project, and it doesn't seem to have worked. It opened ska-sa/spead2#291, where the top of the file indicates Python 3.12, but it's pulled in packages that the requirements.in indicates are only for use with <3.12. |
|
It's deployed already. Can you open a separate issue? Handling markers may require extra tricks, not sure. |
Since https://github.com/jazzband/pip-tools/releases/tag/6.2.0 pip-compile logs Python version (up to minor version) in the generated file, for example
This file is autogenerated by pip-compile with python 3.6.This information could be used to deduce which Python version to use when executing pip-compile before falling back to the default one. I think it has higher accuracy than trying to read
python_versionmarker from generated file because usingpython_versionmarker in*.infile does not necessarily mean it gets compiled under a Python version that satisfies the marker.The text was updated successfully, but these errors were encountered: