feat: allow `uv pip install` to fetch requirement file from remote URL #1

ottaviohartman · 2024-02-17T23:03:12Z

Summary

Allow uv pip install to fetch requirements files from a remote URL

TODO

add tests
- mock HTTP calls in tests

Fixes astral-sh#1481

Test Plan

TODO

…ents

…ements

ottaviohartman · 2024-02-17T23:04:59Z

crates/requirements-txt/src/lib.rs

+            RequirementsTxtSource::Url(url) => {
+                // Use the `reqwest` crate to download the file.
+                reqwest::blocking::get(url.as_str())
+                    .map_err(|err| RequirementsTxtFileError {
+                        file: PathBuf::from(url.as_str()),
+                        error: RequirementsTxtParserError::IO(io::Error::new(
+                            io::ErrorKind::Other,
+                            err.to_string(),
+                        )),
+                    })?
+                    .error_for_status()
+                    .map_err(|err| RequirementsTxtFileError {
+                        file: PathBuf::from(url.as_str()),
+                        error: RequirementsTxtParserError::IO(io::Error::new(
+                            io::ErrorKind::Other,
+                            err.to_string(),
+                        )),
+                    })?
+                    .text()
+                    .map_err(|err| RequirementsTxtFileError {
+                        file: PathBuf::from(url.as_str()),
+                        error: RequirementsTxtParserError::IO(io::Error::new(
+                            io::ErrorKind::Other,
+                            err.to_string(),
+                        )),
+                    })?
+            }


This feels like the wrong place to do this. Should I add a dependency on uv-client and use that instead?

Additionally, this errors with tokio complaining:

Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.

ottaviohartman · 2024-02-17T23:05:51Z

crates/requirements-txt/src/lib.rs

+    #[test]
+    fn url_test() -> Result<()> {
+        let temp_dir = assert_fs::TempDir::new()?;
+        let error = RequirementsTxt::parse("https://example.com/requirements.txt", temp_dir.path())
+            .unwrap_err();
+        let errors = anyhow::Error::new(error).chain().join("\n");
+
+        insta::assert_display_snapshot!(errors, @r###"
+        Unsupported URL (expected a `file://` scheme) in `<REQUIREMENTS_TXT>`: `https://example.com/requirements.txt`
+        "###);
+
+        Ok(())
+    }
+


TODO: add tests

crates/requirements-txt/src/lib.rs

## Summary We had some users report bugs because the Python querying failed due to warnings in `stderr`. I don't think this should fail on any `stderr` output. E.g. ``` × Querying Python at `USER/.pixi/envs/default/bin/python3.10` failed with status exit status: 0 with exit status: 0 │ --- stdout: │ {"markers": {"implementation_name": "cpython", "implementation_version": "3.10.0", "os_name": "posix", "platform_machine": "x86_64", "platform_python_implementation": "CPython", "platform_release": "5.15.146.1-microsoft-standard-WSL2", │ "platform_system": "Linux", "platform_version": "#1 SMP Thu Jan 11 04:09:03 UTC 2024", "python_full_version": "3.10.0", "python_version": "3.10", "sys_platform": "linux"}, "base_prefix": "USER/.pixi/ │ envs/default", "base_exec_prefix": "USER/.pixi/envs/default", "prefix": "USER/.pixi/envs/default", "base_executable": "USER/.pixi/envs/default/ │ bin/python3.10", "sys_executable": "USER/.pixi/envs/default/bin/python3.10", "stdlib": "USER/.pixi/envs/default/lib/python3.10", "scheme": {"platlib": "/home/mvanniekerk/ │ code/vice-python/.pixi/envs/default/lib/python3.10/site-packages", "purelib": "USER/.pixi/envs/default/lib/python3.10/site-packages", "include": "USER/.pixi/envs/default/ │ include/python3.10", "scripts": "USER/.pixi/envs/default/bin", "data": "USER/.pixi/envs/default"}, "virtualenv": {"purelib": "lib/python3.10/site-packages", "platlib": "lib/ │ python3.10/site-packages", "include": "include/site/python3.10", "scripts": "bin", "data": ""}} │ --- stderr: │ [03/21/24 15:59:48] WARNING pyproject.toml does not contain a setuptools.py:119 │ tool.setuptools_scm section │ --- ```

This alias is required for the 32-bit x86 manylinux builder. I've also changed the reporting level for unusable python interpreter errors to debug so it shows up with `-v`, otherwise we can't see what's broken: ```console # uv venv py312 --python python3.12 -vv 0.000461s DEBUG uv_interpreter::discovery Searching for Python 3.12 in search path × No interpreter found for Python 3.12 in search path ``` Reproduction: ```console $ docker run --rm -it quay.io/pypa/manylinux2014_i686 # curl -LsSf https://astral.sh/uv/install.sh | s # source $HOME/.cargo/env # RUST_LOG=trace uv venv py312 --python python3.12 DEBUG Searching for Python 3.12 in search path TRACE Searching PATH for executables: python3.12, python3, python TRACE Checking `PATH` directory for interpreters: /root/.cargo/bin TRACE Checking `PATH` directory for interpreters: /opt/rh/devtoolset-10/root/usr/bin TRACE Checking `PATH` directory for interpreters: /usr/local/sbin TRACE Checking `PATH` directory for interpreters: /usr/local/bin TRACE Found possible Python executable: /usr/local/bin/python3.12 TRACE Querying interpreter executable at /usr/local/bin/python3.12 TRACE Querying Python at `/usr/local/bin/python3.12` did not return the expected data unknown variant `i686`, expected one of `aarch64`, `armv6l`, `armv7l`, `powerpc64le`, `powerpc64`, `x86`, `x86_64`, `s390x` --- stdout: {"result": "success", "markers": {"implementation_name": "cpython", "implementation_version": "3.12.3", "os_name": "posix", "platform_machine": "i686", "platform_python_implementation": "CPython", "platform_release": "6.1.0-10-amd64", "platform_system": "Linux", "platform_version": "#1 SMP PREEMPT_DYNAMIC Debian 6.1.37-1 (2023-07-03)", "python_full_version": "3.12.3", "python_version": "3.12", "sys_platform": "linux"}, "base_prefix": "/opt/_internal/cpython-3.12.3", "base_exec_prefix": "/opt/_internal/cpython-3.12.3", "prefix": "/opt/_internal/cpython-3.12.3", "base_executable": "/usr/local/bin/python3.12", "sys_executable": "/usr/local/bin/python3.12", "sys_path": ["/root/.cache/uv/.tmp7k64ZY", "/opt/_internal/cpython-3.12.3/lib/python312.zip", "/opt/_internal/cpython-3.12.3/lib/python3.12", "/opt/_internal/cpython-3.12.3/lib/python3.12/lib-dynload", "/opt/_internal/cpython-3.12.3/lib/python3.12/site-packages"], "stdlib": "/opt/_internal/cpython-3.12.3/lib/python3.12", "scheme": {"platlib": "/opt/_internal/cpython-3.12.3/lib/python3.12/site-packages", "purelib": "/opt/_internal/cpython-3.12.3/lib/python3.12/site-packages", "include": "/opt/_internal/cpython-3.12.3/include/python3.12", "scripts": "/opt/_internal/cpython-3.12.3/bin", "data": "/opt/_internal/cpython-3.12.3"}, "virtualenv": {"purelib": "lib/python3.12/site-packages", "platlib": "lib/python3.12/site-packages", "include": "include/site/python3.12", "scripts": "bin", "data": ""}, "platform": {"os": {"name": "manylinux", "major": 2, "minor": 17}, "arch": "i686"}, "gil_disabled": false, "pointer_size": "32"} --- stderr: --- TRACE Skipping bad interpreter at /usr/local/bin/python3.12 TRACE Checking `PATH` directory for interpreters: /usr/sbin TRACE Checking `PATH` directory for interpreters: /usr/bin TRACE Found possible Python executable: /usr/bin/python TRACE Querying interpreter executable at /usr/bin/python TRACE Can't use Python at `/usr/bin/python` TRACE Skipping bad interpreter at /usr/bin/python TRACE Checking `PATH` directory for interpreters: /sbin TRACE Checking `PATH` directory for interpreters: /bin TRACE Found possible Python executable: /bin/python TRACE Querying interpreter executable at /bin/python TRACE Can't use Python at `/bin/python` TRACE Skipping bad interpreter at /bin/python ```

## Summary Same as astral-sh#3899 but for ppc64le, there were no tests added there so I wouldn't know where to begin to do so. Using image docker image `quay.io/pypa/manylinux2014_ppc64le` (uv 0.2.4) ``` [root@e3ff544d1337 ~]# python3.12 -V Python 3.12.3 [root@e3ff544d1337 ~]# RUST_LOG="trace" uv venv py312 --python python3.12 DEBUG Searching for Python 3.12 in search path TRACE Searching PATH for executables: python3.12, python3, python TRACE Checking `PATH` directory for interpreters: /opt/rh/devtoolset-10/root/usr/bin TRACE Checking `PATH` directory for interpreters: /usr/local/sbin TRACE Checking `PATH` directory for interpreters: /usr/local/bin TRACE Found possible Python executable: /usr/local/bin/python3.12 TRACE Querying interpreter executable at /usr/local/bin/python3.12 TRACE Querying Python at `/usr/local/bin/python3.12` did not return the expected data unknown variant `ppc64le`, expected one of `aarch64`, `armv6l`, `armv7l`, `powerpc64le`, `powerpc64`, `x86`, `x86_64`, `s390x` --- stdout: {"result": "success", "markers": {"implementation_name": "cpython", "implementation_version": "3.12.3", "os_name": "posix", "platform_machine": "ppc64le", "platform_python_implementation": "CPython", "platform_release": "6.6.26-linuxkit", "platform_system": "Linux", "platform_version": "#1 SMP Sat Apr 27 04:13:19 UTC 2024", "python_full_version": "3.12.3", "python_version": "3.12", "sys_platform": "linux"}, "base_prefix": "/opt/_internal/cpython-3.12.3", "base_exec_prefix": "/opt/_internal/cpython-3.12.3", "prefix": "/opt/_internal/cpython-3.12.3", "base_executable": "/usr/local/bin/python3.12", "sys_executable": "/usr/local/bin/python3.12", "sys_path": ["/root/.cache/uv/.tmpBnM4PN", "/opt/_internal/cpython-3.12.3/lib/python312.zip", "/opt/_internal/cpython-3.12.3/lib/python3.12", "/opt/_internal/cpython-3.12.3/lib/python3.12/lib-dynload", "/opt/_internal/cpython-3.12.3/lib/python3.12/site-packages"], "stdlib": "/opt/_internal/cpython-3.12.3/lib/python3.12", "scheme": {"platlib": "/opt/_internal/cpython-3.12.3/lib/python3.12/site-packages", "purelib": "/opt/_internal/cpython-3.12.3/lib/python3.12/site-packages", "include": "/opt/_internal/cpython-3.12.3/include/python3.12", "scripts": "/opt/_internal/cpython-3.12.3/bin", "data": "/opt/_internal/cpython-3.12.3"}, "virtualenv": {"purelib": "lib/python3.12/site-packages", "platlib": "lib/python3.12/site-packages", "include": "include/site/python3.12", "scripts": "bin", "data": ""}, "platform": {"os": {"name": "manylinux", "major": 2, "minor": 17}, "arch": "ppc64le"}, "gil_disabled": false, "pointer_size": "64"} --- stderr: --- TRACE Skipping bad interpreter at /usr/local/bin/python3.12 TRACE Checking `PATH` directory for interpreters: /usr/sbin TRACE Checking `PATH` directory for interpreters: /usr/bin TRACE Found possible Python executable: /usr/bin/python TRACE Querying interpreter executable at /usr/bin/python TRACE Can't use Python at `/usr/bin/python` TRACE Skipping bad interpreter at /usr/bin/python TRACE Checking `PATH` directory for interpreters: /sbin TRACE Checking `PATH` directory for interpreters: /bin TRACE Found possible Python executable: /bin/python TRACE Querying interpreter executable at /bin/python TRACE Can't use Python at `/bin/python` TRACE Skipping bad interpreter at /bin/python × No interpreter found for Python 3.12 in search path ``` --------- Co-authored-by: Charlie Marsh <crmarsh416@gmail.com>

ottaviohartman added 16 commits February 17, 2024 10:56

init

8e902e9

clippy

6a26bbf

Merge remote-tracking branch 'origin/main' into thartman/url-requirem…

198aeee

…ents

builds

ccf75c8

remove println

b20d633

move

addc6b8

compiles and tests pass

15b3c6c

clippY'

158ccd2

actually working

3f14d86

this almost works

6acade8

this almost works

d0dd418

Requiremenstxt(String0 and PyprojectToml(String)

90a755a

working?

61d1f22

more

7e44aa0

&PathBuf -> &Path

daac37a

Merge branch 'main' of github.com:omh1280/uv into thartman/url-requir…

ccc1c3f

…ements

ottaviohartman commented Feb 17, 2024

View reviewed changes

crates/requirements-txt/src/lib.rs Outdated Show resolved Hide resolved

ottaviohartman added 5 commits February 17, 2024 18:13

nit

a6e6364

nit

c9b47af

docs

fca7813

rearrange

c1f9cd8

fix tests

c994494

ottaviohartman closed this Feb 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: allow `uv pip install` to fetch requirement file from remote URL #1

feat: allow `uv pip install` to fetch requirement file from remote URL #1

ottaviohartman commented Feb 17, 2024

ottaviohartman Feb 17, 2024

ottaviohartman Feb 17, 2024

feat: allow uv pip install to fetch requirement file from remote URL #1

feat: allow uv pip install to fetch requirement file from remote URL #1

Conversation

ottaviohartman commented Feb 17, 2024

Summary

TODO

Test Plan

ottaviohartman Feb 17, 2024

Choose a reason for hiding this comment

ottaviohartman Feb 17, 2024

Choose a reason for hiding this comment

feat: allow `uv pip install` to fetch requirement file from remote URL #1

feat: allow `uv pip install` to fetch requirement file from remote URL #1