Cannot download some package

[ZuluPro]

With a brand-new docker install, I cannot download some package.
The latest version raise a 404 for .metadata

Reproduction

Here's some example with Sphinx==7.0.1:

$ pip install sphinx==7.0.1 --index-url=http://proxpi:5000/index/ --ignore-installed --no-cache
Looking in indexes: http://proxpi:5000/index/
Collecting sphinx==7.0.1
  Obtaining dependency information for sphinx==7.0.1 from http://proxpi:5000/index/sphinx/sphinx-7.0.1-py3-none-any.whl.metadata
  ERROR: HTTP error 404 while getting http://proxpi:5000/index/sphinx/sphinx-7.0.1-py3-none-any.whl.metadata
ERROR: 404 Client Error: Not Found for url: http://proxpi:5000/index/sphinx/sphinx-7.0.1-py3-none-any.whl.metadata

With the original repo it works:

$ pip install sphinx==7.0.1 --index-url=https://pypi.org/simple/
Looking in indexes: https://pypi.org/simple/
...

Also previous versions of Sphinx works:

$ pip install sphinx==7.0.0 --index-url=http://proxpi:5000/index/ --ignore-installed --no-cache
Looking in indexes: http://proxpi:5000/index/
Collecting sphinx==7.0.0
  Downloading http://proxpi.cloud-mercato.lan:5051/index/sphinx/sphinx-7.0.0-py3-none-any.whl (3.0 MB)
...

Expected behaviour

I should be able to install the last package transparently

Actual behaviour

Include current behaviour and output.

Environment

• proxpi: latest
• Environment: Docker container
• Python: 3.10.6
• OS: macOS M2 Ventura

[hzrari]

Hi,
Same here with setuptools

pip install --index-url http://proxpi:15000/index/ --trusted-host proxpi setuptools
Looking in indexes: http://proxpi:15000/index/
Collecting setuptools
  Obtaining dependency information for setuptools from http://proxpi:15000/index/setuptools/setuptools-67.8.0-py3-none-any.whl.metadata
  ERROR: HTTP error 404 while getting http://proxpi:15000/index/setuptools/setuptools-67.8.0-py3-none-any.whl.metadata
ERROR: 404 Client Error: NOT FOUND for url: http://proxpi:15000/index/setuptools/setuptools-67.8.0-py3-none-any.whl.metadata

Whether an older one is downloaded successfully:

pip download --index-url http://proxpi:15000/index/ --trusted-host proxpi setuptools==67.0.0
Looking in indexes: http://proxpi:15000/index/
Collecting setuptools==67.0.0
  Downloading http://proxpi:15000/index/setuptools/setuptools-67.0.0-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 394.7 kB/s eta 0:00:00
Saved ./setuptools-67.0.0-py3-none-any.whl
Successfully downloaded setuptools

On the server side i have these logs:

2023-05-20 11:09:18,287 [    INFO] gunicorn.access: 172.18.0.1 "GET /index/setuptools/setuptools-67.0.0-py3-none-any.whl HTTP/1.1" 200 0 1ms
2023-05-20 11:11:17,711 [    INFO] gunicorn.access: 172.18.0.1 "GET /index/setuptools/setuptools-67.8.0-py3-none-any.whl.metadata HTTP/1.1" 404 207 1ms

I am still digging on my side

[hzrari]

I am still debugging on the pip side (client) to understand how it works:

I modified this file locally in /usr/local/lib/python3.11/site-packages/pip/_internal/operations/prepare.py.
I just forced this function to stop :

    def _fetch_metadata_using_link_data_attr(
        self,
        req: InstallRequirement,
    ) -> Optional[BaseDistribution]:
        """Fetch metadata from the data-dist-info-metadata attribute, if possible."""
        # (1) Get the link to the metadata file, if provided by the backend.
        metadata_link = req.link.metadata_link()
        metadata_link = None
        if metadata_link is None:
            return None
        assert req.req is not None
        logger.info(
            "Obtaining dependency information for %s from %s",
            req.req,
            metadata_link,
        )
        # (2) Download the contents of the METADATA file, separate from the dist itself.
        metadata_file = get_http_url(
            metadata_link,
            self._download,
            hashes=metadata_link.as_hashes(),
        )
        with open(metadata_file.path, "rb") as f:
  

My line is here:
metadata_link = None

Now i am able to download my package successfully:

pip download --index-url http://proxpi:15000/index/ --trusted-host proxpi setuptools==67.8.0 --no-cache
Looking in indexes: http://proxpi:15000/index/
Collecting setuptools==67.8.0
  Downloading http://proxpi:15000/index/setuptools/setuptools-67.8.0-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 389.7 kB/s eta 0:00:00
Saved ./setuptools-67.8.0-py3-none-any.whl
Successfully downloaded setuptools

It seems that some packages are saying that they should expose a kind of metadata file (a kind of description file) but proxpi is not mirroring it.

I followed the function here _internal/models/link.py (always in pip client side):

    def metadata_link(self) -> Optional["Link"]:
        """Implementation of PEP 658 parsing."""
        # Note that Link.from_element() parsing the "data-dist-info-metadata" attribute
        # from an HTML anchor tag is typically how the Link.dist_info_metadata attribute
        # gets set.
        if self.dist_info_metadata is None:
            return None
        metadata_url = f"{self.url_without_fragment}.metadata"
        metadata_link_hash = LinkHash.parse_pep658_hash(self.dist_info_metadata)
        if metadata_link_hash is None:
            return Link(metadata_url)
        return Link(metadata_url, hashes=metadata_link_hash.as_dict())

So indeed, some packages seems now to support pep658 (https://peps.python.org/pep-0658/#specification) in their build process.

This is confirmed by my command:
curl -H 'Accept: application/vnd.pypi.simple.v1+html' http://proxpi:15000/index/setuptools/

We can see these lines:

    <a href="setuptools-67.7.1-py3-none-any.whl#sha256=6f0839fbdb7e3cfef1fc38d7954f5c1c26bf4eebb155a55c9bf8faf997b9fb67" data-requires-python="&gt;=3.7">setuptools-67.7.1-py3-none-any.whl</a><br />
    <a href="setuptools-67.7.1.tar.gz#sha256=bb16732e8eb928922eabaa022f881ae2b7cdcfaf9993ef1f5e841a96d32b8e0c" data-requires-python="&gt;=3.7">setuptools-67.7.1.tar.gz</a><br />
    <a href="setuptools-67.7.2-py3-none-any.whl#sha256=23aaf86b85ca52ceb801d32703f12d77517b2556af839621c641fca11287952b" data-requires-python="&gt;=3.7">setuptools-67.7.2-py3-none-any.whl</a><br />
    <a href="setuptools-67.7.2.tar.gz#sha256=f104fa03692a2602fa0fec6c6a9e63b6c8a968de13e17c026957dd1f53d80990" data-requires-python="&gt;=3.7">setuptools-67.7.2.tar.gz</a><br />
    <a href="setuptools-67.8.0-py3-none-any.whl#sha256=5df61bf30bb10c6f756eb19e7c9f3b473051f48db77fddbe06ff2ca307df9a6f" data-requires-python="&gt;=3.7" data-dist-info-metadata="sha256=808ead36e01cd1358387122c08bc4ad4e745296b7a811046ddaa50e7101389af">setuptools-67.8.0-py3-none-any.whl</a><br />
    <a href="setuptools-67.8.0.tar.gz#sha256=62642358adc77ffa87233bc4d2354c4b2682d214048f500964dbe760ccedf102" data-requires-python="&gt;=3.7">setuptools-67.8.0.tar.gz</a><br />

So the format has changed in the last release to respect pep658

I will now try to workaround that by in proxpi. As i am not a proxpi developer, my idea is to simply mirror the same package name (expanded with .metadata) if it it exists in the original repository.

I will let you know

[hzrari]

I changed my mind, because i have found a better/faster workaround, and as i my blocked, i need to go faster. As I understood how proxpi is creating the index for each package, i was able to skip and data-dist-info-metadata from the generated index.

This is my patch:

# diff -u  _cache.py.orig _cache.py 
--- _cache.py.orig
+++ _cache.py
@@ -228,7 +228,7 @@
             if child.tag == "a":
                 name = child.text
                 url = urllib.parse.urljoin(response.request.url, child.attrib["href"])
-                attributes = {k: v for k, v in child.attrib.items() if k != "href"}
+                attributes = {k: v for k, v in child.attrib.items() if k not in ["href", "data-dist-info-metadata"]}
                 fragment = urllib.parse.urlsplit(url).fragment
                 package.files[name] = File(name, url, fragment, attributes)
         self._packages[package_name] = package

If you are using the docker image, do not forget to restart the container to see the change reflected by Gunicorn

Now the same curl give the suitable output:
curl -H 'Accept: application/vnd.pypi.simple.v1+html' http://proxpi:15000/index/setuptools/

output:

<a href="setuptools-67.7.0.tar.gz#sha256=b7e53a01c6c654d26d2999ee033d8c6125e5fa55f03b7b193f937ae7ac999f22" data-requires-python="&gt;=3.7">setuptools-67.7.0.tar.gz</a><br />
  <a href="setuptools-67.7.1-py3-none-any.whl#sha256=6f0839fbdb7e3cfef1fc38d7954f5c1c26bf4eebb155a55c9bf8faf997b9fb67" data-requires-python="&gt;=3.7">setuptools-67.7.1-py3-none-any.whl</a><br />
  <a href="setuptools-67.7.1.tar.gz#sha256=bb16732e8eb928922eabaa022f881ae2b7cdcfaf9993ef1f5e841a96d32b8e0c" data-requires-python="&gt;=3.7">setuptools-67.7.1.tar.gz</a><br />
  <a href="setuptools-67.7.2-py3-none-any.whl#sha256=23aaf86b85ca52ceb801d32703f12d77517b2556af839621c641fca11287952b" data-requires-python="&gt;=3.7">setuptools-67.7.2-py3-none-any.whl</a><br />
  <a href="setuptools-67.7.2.tar.gz#sha256=f104fa03692a2602fa0fec6c6a9e63b6c8a968de13e17c026957dd1f53d80990" data-requires-python="&gt;=3.7">setuptools-67.7.2.tar.gz</a><br />
  <a href="setuptools-67.8.0-py3-none-any.whl#sha256=5df61bf30bb10c6f756eb19e7c9f3b473051f48db77fddbe06ff2ca307df9a6f" data-requires-python="&gt;=3.7">setuptools-67.8.0-py3-none-any.whl</a><br />
  <a href="setuptools-67.8.0.tar.gz#sha256=62642358adc77ffa87233bc4d2354c4b2682d214048f500964dbe760ccedf102" data-requires-python="&gt;=3.7">setuptools-67.8.0.tar.gz</a><br />

I reverted my pip client as it was, and now I am able to download my package:

pip download --index-url http://proxpi:15000/index/ --trusted-host proxpi setuptools==67.8.0 --no-cache
Looking in indexes: http://proxpi:15000/index/
Collecting setuptools==67.8.0
  Downloading http://proxpi:15000/index/setuptools/setuptools-67.8.0-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 396.0 kB/s eta 0:00:00
Saved ./setuptools-67.8.0-py3-none-any.whl
Successfully downloaded setuptools

I still believe that the good solution is to mirror the metadata file, but for blocked people (like me in production) this is the fastest way I found to make the change locally in proxpi.

@ZuluPro you have here a workaround if you are blocked

Thanks to proxpi and to the project maintainer.

[ZuluPro]

Hi @hzrari
Wow! Thank you a lot for this debugging

Do you plan to do a PR ?

[hzrari]

Hello @ZuluPro
I don't think I need to do that, my patch is only a workaround to unlock the situation in this version.
It seems that the final implementation is already done but not release or built yet: e33d32e

It is included in this version: https://github.com/EpicWink/proxpi/releases/tag/v1.1.0rc0
As it is an RC (Release Candidate), we may need to wait for the 1.1.0, or use this docker image:
docker pull epicwink/proxpi:v1.1.0rc0

As we can see on the hub, it is not tagged as latest.
On my side I prefer to patch the version that we have (latest) with only this patch, as it is stable until now, instead of putting in production an RC one. It is up to you to choose the suitable solution for your case.

[EpicWink]

Caused by pypi/warehouse#13649

proxpi v1.1rc0 is simply awaiting community testing: it's been working for me for months. If this fixes your issue (and you don't have any other problems), please react on the release with a rocket-ship 🚀

[hzrari]

ok @EpicWink , Indeed, I have seen indeed the huge work and thank you for that.
You want me to put in Production the v1.1.0rc0 and give you a feedback after a week of use for example ? I think we should have like a 1000 pip install per day which are targeting pour local proxpi.

Regards

[ZuluPro]

As it is fixed in version 1.1.0rc0 : https://github.com/EpicWink/proxpi/releases/tag/v1.1.0rc0

I close this issue.

Thank you guys

[EpicWink]

Note, you may will run into pypa/pip#12042 if your proxied index server correctly provides the dist-info-metadata attribute.

Solve by doing one of:

• removing dist-info-metadata from your index
• modifying proxpi to not provide JSON responses: make _wants_json always return False
• modifying proxpi to not provide dist-info-metadata fields: see proxpi._cache.File.to_json_response

[EpicWink]

If you want to continue using v1.0, I've released a new version: v1.0.2rc0. This only includes the fix from @hzrari above.

[EpicWink]

v1.1 has now been released