Lazy wheel broken in 1.8.0 and 1.8.1 when there are redirects

[thatch]

Description

I'm trying to package up a nice repro, but the gist of it is in a custom simple mirror, we use this convention:

/simple/foo/ includes a url like /packages/<n>/foo-0.1-py3-none-any.whl
/packages/<n>/foo-0.1-py3-none-any.whl gives a 307 to a pythonhosted url

It fails with the following formatted traceback (the wheel url doesn't matter; I had to clear caches a couple of times during testing to be sure it would trigger)

% time poetry lock --no-update
Updating dependencies
Resolving dependencies... (28.2s)
Resolving dependencies... (28.3s)
Resolving dependencies... (28.4s)
Resolving dependencies... (37.9s)
Resolving dependencies... (38.0s)
Resolving dependencies... (38.1s)
Resolving dependencies... (52.7s)

  ValueError

  Package('opentelemetry-instrumentation-celery', '0.25b0') is not in list

  at <redacted>/lib/python3.10/site-packages/poetry/repositories/legacy_repository.py:66 in package
       62│         Note that this will be cached so the subsequent operations
       63│         should be much faster.
       64│         """
       65│         try:
    →  66│             index = self._packages.index(Package(name, version))
       67│ 
       68│             return self._packages[index]
       69│         except ValueError:
       70│             package = super().package(name, version, extras)

The following error occurred when trying to handle this error:


  HTTPError

  501 Server Error: Unsupported client range for url: https://files.pythonhosted.org/packages/67/1b/8920801a4361dd069cc7edf9106ceb308fa9b8adc703f3fcd516195f4a4e/opentelemetry_instrumentation_celery-0.25b0-py3-none-any.whl#sha256=1d9b88e8aa861d1fe0b4bd1fb3b9e5032e7a5b65c5419fcb6977b778e52cf621

  at <redacted>/lib/python3.10/site-packages/requests/models.py:1021 in raise_for_status
      1017│                 f"{self.status_code} Server Error: {reason} for url: {self.url}"
      1018│             )
      1019│ 
      1020│         if http_error_msg:
    → 1021│             raise HTTPError(http_error_msg, response=self)
      1022│ 
      1023│     def close(self):
      1024│         """Releases the connection back to the pool. Once this method has been
      1025│         called the underlying ``raw`` object must not be accessed again.

The following error occurred when trying to handle this error:


  AssertionError

  

  at <redacted>/lib/python3.10/site-packages/poetry/inspection/lazy_wheel.py:388 in _content_length_from_head
      384│                                              for "bytes" ranges."""
      385│         self._request_count += 1
      386│         head = self._session.head(self._url, headers=self._uncached_headers())
      387│         head.raise_for_status()
    → 388│         assert head.status_code == codes.ok
      389│         accepted_range = head.headers.get("Accept-Ranges", None)
      390│         if accepted_range != "bytes":
      391│             raise HTTPRangeRequestUnsupported(
      392│                 f"server does not support byte ranges: header was '{accepted_range}'"
poetry lock --no-update  7.50s user 0.94s system 15% cpu 54.780 total

Workarounds

Downgrade to 1.7.1 if you use redirects in your simple mirror

Poetry Installation Method

pip

Operating System

OS X (does not seem to be OS-dependent)

Poetry Version

Poetry (version 1.8.1)

Poetry Configuration

cache-dir = "/Users/timhatch/Library/Caches/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
repositories.<redacted>.url = "https://<redact>/simple/"
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/timhatch/Library/Caches/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

I don't think this is relevant

Example pyproject.toml

[[tool.poetry.source]]
name = "<redact>"
url = "https://<redacted>/simple/"
priority = "primary"

[[tool.poetry.source]]
name = "PyPI"
priority = "supplemental"

...plus some normal deps...

Poetry Runtime Logs

[urllib3:urllib3.connectionpool] https://<redacted>:443 "GET /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): files.pythonhosted.org:443
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl HTTP/1.1" 501 463
Retrying HTTP request in 0.0 seconds.
[urllib3:urllib3.connectionpool] https://<redacted>:443 "GET /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (2): files.pythonhosted.org:443
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl HTTP/1.1" 501 463
Retrying HTTP request in 0.0 seconds.
[urllib3:urllib3.connectionpool] https://<redacted>:443 "GET /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (3): files.pythonhosted.org:443
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl HTTP/1.1" 501 465
Retrying HTTP request in 0.0 seconds.
[urllib3:urllib3.connectionpool] https://<redacted>:443 "GET /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (4): files.pythonhosted.org:443
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl HTTP/1.1" 501 462
Retrying HTTP request in 0.0 seconds.
[urllib3:urllib3.connectionpool] https://<redacted>:443 "GET /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (5): files.pythonhosted.org:443
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl HTTP/1.1" 501 465
Retrying HTTP request in 0.0 seconds.
[urllib3:urllib3.connectionpool] https://<redacted>:443 "GET /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (6): files.pythonhosted.org:443
[urllib3:urllib3.connectionpool] https://files.pythonhosted.org:443 "GET /packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl HTTP/1.1" 501 463
Negative byte range not supported for domain '<redacted>': using HEAD request before lazy wheel from now on (code: 501)
[urllib3:urllib3.connectionpool] https://<redacted>:443 "HEAD /packages/8515241215/responses-0.25.0-py3-none-any.whl HTTP/1.1" 307 0
   1: Version solving took 14.982 seconds.
   1: Tried 1 solutions.

  ValueError

  Package('responses', '0.25.0') is not in list

  at .venv/lib/python3.11/site-packages/poetry/repositories/legacy_repository.py:66 in package
       62│         Note that this will be cached so the subsequent operations
       63│         should be much faster.
       64│         """
       65│         try:
    →  66│             index = self._packages.index(Package(name, version))
       67│ 
       68│             return self._packages[index]
       69│         except ValueError:
       70│             package = super().package(name, version, extras)

The following error occurred when trying to handle this error:


  Stack trace:

  4  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:609 in _extract_content_length
      607│         try:
      608│             # Initial range request for just the end of the file.
    → 609│             file_length, tail = self._try_initial_chunk_request(initial_chunk_size)
      610│         except HTTPError as e:
      611│             # Our initial request using a negative byte range was not supported.

  3  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:572 in _try_initial_chunk_request
      570│ 
      571│         self._request_count += 1
    → 572│         tail = self._session.get(self._url, headers=headers, stream=True)
      573│         tail.raise_for_status()
      574│ 

  2  .venv/lib/python3.11/site-packages/poetry/utils/authenticator.py:267 in get
      265│ 
      266│     def get(self, url: str, **kwargs: Any) -> requests.Response:
    → 267│         return self.request("get", url, **kwargs)
      268│ 
      269│     def head(self, url: str, **kwargs: Any) -> requests.Response:

  1  .venv/lib/python3.11/site-packages/poetry/utils/authenticator.py:245 in request
      243│                 if resp.status_code not in STATUS_FORCELIST or is_last_attempt:
      244│                     if raise_for_status:
    → 245│                         resp.raise_for_status()
      246│                     return resp
      247│ 

  HTTPError

  501 Server Error: Unsupported client range for url: https://files.pythonhosted.org/packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl#sha256=2f0b9c2b6437db4b528619a77e5d565e4ec2a9532162ac1a131a83529db7be1a

  at .venv/lib/python3.11/site-packages/requests/models.py:1021 in raise_for_status
      1017│                 f"{self.status_code} Server Error: {reason} for url: {self.url}"
      1018│             )
      1019│ 
      1020│         if http_error_msg:
    → 1021│             raise HTTPError(http_error_msg, response=self)
      1022│ 
      1023│     def close(self):
      1024│         """Releases the connection back to the pool. Once this method has been
      1025│         called the underlying ``raw`` object must not be accessed again.

The following error occurred when trying to handle this error:


  Stack trace:

  31  .venv/lib/python3.11/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  30  .venv/lib/python3.11/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

  29  .venv/lib/python3.11/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

  28  .venv/lib/python3.11/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

  27  .venv/lib/python3.11/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  26  .venv/lib/python3.11/site-packages/cleo/commands/base_command.py:117 in run
       115│         io.input.validate()
       116│ 
     → 117│         return self.execute(io) or 0
       118│ 
       119│     def merge_application_definition(self, merge_args: bool = True) -> None:

  25  .venv/lib/python3.11/site-packages/cleo/commands/command.py:61 in execute
        59│ 
        60│         try:
     →  61│             return self.handle()
        62│         except KeyboardInterrupt:
        63│             return 1

  24  .venv/lib/python3.11/site-packages/poetry/console/commands/lock.py:55 in handle
        53│         self.installer.lock(update=not self.option("no-update"))
        54│ 
     →  55│         return self.installer.run()
        56│ 

  23  .venv/lib/python3.11/site-packages/poetry/installation/installer.py:104 in run
       102│             self.verbose(True)
       103│ 
     → 104│         return self._do_install()
       105│ 
       106│     def dry_run(self, dry_run: bool = True) -> Installer:

  22  .venv/lib/python3.11/site-packages/poetry/installation/installer.py:241 in _do_install
       239│                 source_root=self._env.path.joinpath("src")
       240│             ):
     → 241│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       242│         else:
       243│             self._io.write_line("Installing dependencies from lock file")

  21  .venv/lib/python3.11/site-packages/poetry/puzzle/solver.py:71 in solve
        69│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        70│             start = time.time()
     →  71│             packages, depths = self._solve()
        72│             end = time.time()
        73│ 

  20  .venv/lib/python3.11/site-packages/poetry/puzzle/solver.py:154 in _solve
       152│ 
       153│         try:
     → 154│             result = resolve_version(self._package, self._provider)
       155│ 
       156│             packages = result.packages

  19  .venv/lib/python3.11/site-packages/poetry/mixology/__init__.py:18 in resolve_version
        16│     solver = VersionSolver(root, provider)
        17│ 
     →  18│     return solver.solve()
        19│ 

  18  .venv/lib/python3.11/site-packages/poetry/mixology/version_solver.py:175 in solve
       173│             while next is not None:
       174│                 self._propagate(next)
     → 175│                 next = self._choose_package_version()
       176│ 
       177│             return self._result()

  17  .venv/lib/python3.11/site-packages/poetry/mixology/version_solver.py:514 in _choose_package_version
       512│             package = locked
       513│ 
     → 514│         package = self._provider.complete_package(package)
       515│ 
       516│         conflict = False

  16  .venv/lib/python3.11/site-packages/poetry/puzzle/provider.py:489 in complete_package
       487│                 dependency_package = DependencyPackage(
       488│                     dependency,
     → 489│                     self._pool.package(
       490│                         package.pretty_name,
       491│                         package.version,

  15  .venv/lib/python3.11/site-packages/poetry/repositories/repository_pool.py:204 in package
       202│         for repo in self.repositories:
       203│             try:
     → 204│                 return repo.package(name, version, extras=extras)
       205│             except PackageNotFound:
       206│                 continue

  14  .venv/lib/python3.11/site-packages/poetry/repositories/legacy_repository.py:70 in package
        68│             return self._packages[index]
        69│         except ValueError:
     →  70│             package = super().package(name, version, extras)
        71│             package._source_type = "legacy"
        72│             package._source_url = self._url

  13  .venv/lib/python3.11/site-packages/poetry/repositories/cached_repository.py:75 in package
        73│         extras: list[str] | None = None,
        74│     ) -> Package:
     →  75│         return self.get_release_info(canonicalize_name(name), version).to_package(
        76│             name=name, extras=extras
        77│         )

  12  .venv/lib/python3.11/site-packages/poetry/repositories/cached_repository.py:52 in get_release_info
        50│             return PackageInfo.load(self._get_release_info(name, version))
        51│ 
     →  52│         cached = self._release_cache.remember(
        53│             f"{name}:{version}", lambda: self._get_release_info(name, version)
        54│         )

  11  .venv/lib/python3.11/site-packages/poetry/utils/cache.py:147 in remember
       145│         value = self.get(key)
       146│         if value is None:
     → 147│             value = callback() if callable(callback) else callback
       148│             self.put(key, value, minutes)
       149│         return value

  10  .venv/lib/python3.11/site-packages/poetry/repositories/cached_repository.py:53 in <lambda>
        51│ 
        52│         cached = self._release_cache.remember(
     →  53│             f"{name}:{version}", lambda: self._get_release_info(name, version)
        54│         )
        55│ 

   9  .venv/lib/python3.11/site-packages/poetry/repositories/legacy_repository.py:123 in _get_release_info
       121│         yanked = page.yanked(name, version)
       122│ 
     → 123│         return self._links_to_data(
       124│             links,
       125│             PackageInfo(

   8  .venv/lib/python3.11/site-packages/poetry/repositories/http_repository.py:360 in _links_to_data
       358│ 
       359│         # drop yanked files unless the entire release is yanked
     → 360│         info = self._get_info_from_links(links, ignore_yanked=not data.yanked)
       361│ 
       362│         data.summary = info.summary

   7  .venv/lib/python3.11/site-packages/poetry/repositories/http_repository.py:310 in _get_info_from_links
       308│                 return self._get_info_from_metadata(
       309│                     universal_python3_wheel
     → 310│                 ) or self._get_info_from_wheel(universal_python3_wheel)
       311│ 
       312│             if universal_python2_wheel:

   6  .venv/lib/python3.11/site-packages/poetry/repositories/http_repository.py:123 in _get_info_from_wheel
       121│             try:
       122│                 package_info = PackageInfo.from_metadata(
     → 123│                     metadata_from_wheel_url(link.filename, link.url, self.session)
       124│                 )
       125│             except HTTPRangeRequestUnsupported:

   5  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:77 in metadata_from_wheel_url
        75│         # After context manager exit, wheel.name will point to a deleted file path.
        76│         # Add `delete_backing_file=False` to disable this for debugging.
     →  77│         with LazyWheelOverHTTP(url, session) as lazy_file:
        78│             metadata_bytes = lazy_file.read_metadata(name)
        79│ 

   4  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:289 in __enter__
       287│     def __enter__(self: U) -> U:
       288│         super().__enter__()
     → 289│         self._setup_content()
       290│         return self
       291│ 

   3  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:357 in _setup_content
       355│         if self._length is None:
       356│             logger.debug("begin fetching content length")
     → 357│             self._length = self._fetch_content_length()
       358│             logger.debug("done fetching content length (is: %d)", self._length)
       359│             # Enable us to seek and write anywhere in the backing file up to this

   2  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:508 in _fetch_content_length
       506│         """
       507│         initial_chunk_size = self._initial_chunk_length()
     → 508│         ret_length, tail = self._extract_content_length(initial_chunk_size)
       509│ 
       510│         # Need to explicitly truncate here in order to perform the write and seek

   1  .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:638 in _extract_content_length
       636│             self._domains_without_negative_range.add(domain)
       637│             # Apply a HEAD request to get the real size, and nothing else for now.
     → 638│             return self._content_length_from_head(), None
       639│ 
       640│         # Some servers that do not support negative offsets,

  AssertionError

  

  at .venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:388 in _content_length_from_head
      384│                                              for "bytes" ranges."""
      385│         self._request_count += 1
      386│         head = self._session.head(self._url, headers=self._uncached_headers())
      387│         head.raise_for_status()
    → 388│         assert head.status_code == codes.ok
      389│         accepted_range = head.headers.get("Accept-Ranges", None)
      390│         if accepted_range != "bytes":
      391│             raise HTTPRangeRequestUnsupported(
      392│                 f"server does not support byte ranges: header was '{accepted_range}'"

[thatch]

Simplified repro:

>>> from poetry.inspection.lazy_wheel import metadata_from_wheel_url
>>> from requests import Session
>>> metadata_from_wheel_url("responses", "https://httpbin.org/redirect-to?url=https://files.pythonhosted.org/packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl", Session())
Traceback (most recent call last):
  File ".venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py", line 609, in _extract_content_length
    file_length, tail = self._try_initial_chunk_request(initial_chunk_size)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py", line 573, in _try_initial_chunk_request
    tail.raise_for_status()
  File ".venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 501 Server Error: Unsupported client range for url: https://files.pythonhosted.org/packages/30/0b/bff1e6a5b646e6ff770deb6a292a96bd844ea13fb523ccbd9209fc4b90b8/responses-0.25.0-py3-none-any.whl

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py", line 77, in metadata_from_wheel_url
    with LazyWheelOverHTTP(url, session) as lazy_file:
  File ".venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py", line 289, in __enter__
    self._setup_content()
  File ".venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py", line 357, in _setup_content
    self._length = self._fetch_content_length()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py", line 508, in _fetch_content_length
    ret_length, tail = self._extract_content_length(initial_chunk_size)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py", line 638, in _extract_content_length
    return self._content_length_from_head(), None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py", line 388, in _content_length_from_head
    assert head.status_code == codes.ok
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

[dimbleby]

perhaps just allow_redirects=True on the HEAD request

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.