AttributeError __spec__ on Python 3.11

[pawamoy]

Running pytest on Python 3.11:

% python3.11 -m pytest -c config/pytest.ini -n auto tests
Traceback (most recent call last):
  File "/home/pawamoy/.cache/pdm/packages/py-1.10.0-py2.py3-none-any/lib/py/_vendored_packages/apipkg/__init__.py", line 145, in __makeattr
    modpath, attrname = self.__map__[name]
                        ~~~~~~~~~~~~^^^^^^
KeyError: '__spec__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pawamoy/.basher-packages/pyenv/pyenv/versions/3.11-dev/lib/python3.11/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/.basher-packages/pyenv/pyenv/versions/3.11-dev/lib/python3.11/runpy.py", line 146, in _get_module_details
    return _get_module_details(pkg_main_name, error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/.basher-packages/pyenv/pyenv/versions/3.11-dev/lib/python3.11/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
    ^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/.cache/pdm/packages/pytest-6.2.5-py3-none-any/lib/pytest/__init__.py", line 5, in <module>
    from _pytest.assertion import register_assert_rewrite
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/.cache/pdm/packages/pytest-6.2.5-py3-none-any/lib/_pytest/assertion/__init__.py", line 9, in <module>
    from _pytest.assertion import rewrite
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/.cache/pdm/packages/pytest-6.2.5-py3-none-any/lib/_pytest/assertion/rewrite.py", line 34, in <module>
    from _pytest.assertion import util
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/.cache/pdm/packages/pytest-6.2.5-py3-none-any/lib/_pytest/assertion/util.py", line 13, in <module>
    import _pytest._code
    ^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/.cache/pdm/packages/pytest-6.2.5-py3-none-any/lib/_pytest/_code/__init__.py", line 2, in <module>
    from .code import Code
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/.cache/pdm/packages/pytest-6.2.5-py3-none-any/lib/_pytest/_code/code.py", line 54, in <module>
    class Code:
    ^^^^^^^^^^^
  File "/home/pawamoy/.cache/pdm/packages/pytest-6.2.5-py3-none-any/lib/_pytest/_code/code.py", line 81, in Code
    def path(self) -> Union[py.path.local, str]:
                            ^^^^^^^^^^^^^
  File "/home/pawamoy/.cache/pdm/packages/py-1.10.0-py2.py3-none-any/lib/py/_vendored_packages/apipkg/__init__.py", line 152, in __makeattr
    result = importobj(modpath, attrname)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pawamoy/.cache/pdm/packages/py-1.10.0-py2.py3-none-any/lib/py/_vendored_packages/apipkg/__init__.py", line 72, in importobj
    module = __import__(modpath, None, None, ['__doc__'])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1044, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
  File "/home/pawamoy/.cache/pdm/packages/py-1.10.0-py2.py3-none-any/lib/py/_vendored_packages/apipkg/__init__.py", line 150, in __makeattr
    raise AttributeError(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: __spec__

Note: I'm using PDM which uses .pth files in __pypackages__ that point to actual installed packages in ~/.cache/pdm/packages (it allows dedup of packages across projects, saving space and installation time). Not sure this has to do with the issue, but I hear py/apipkg do some black magic on themselves, so this might conflict. It works well on other Python versions though (3.8, 3.9 and 3.10).

[RonnyPfannschmidt]

This may require a coordinated release of py and apipkg, im not sure if I have the near term capacity

If the pending apipkg 2 can work, ill haste the release

Else i need to find time/motivation to fix apipkg for python 3.11

[pawamoy]

Well there's no rush of course, as Python 3.11 is not gonna be released soon 🙂
If you can give directions on how to tackle this, maybe someone else will be willing to do it!

[RonnyPfannschmidt]

First a verification of standalone apipkg is needed, then after fix it has to be included in the vendored items

[greyblue9]

The cause of this error I tracked down to this change (commit 0a8ae8a50a0fea3d39ec49b220a5c7a5b70e36f8 in cpython):
'bpo-44717: improve AttributeError on circular imports of submodules (GH-27338)'
https://bugs.python.org/issue44717

it introduces the parent = parent_module.__spec__ line in importlib.

AttributeError: __spec__ (which is thrown from importlib, while KeyError in apipkg is being handled)
The latest master and modernization branches fix the issue in pytest (and apipkg standalone) for me.

From the stack in importlib:

File /data/media/0/src/python3/src/Lib/importlib/_bootstrap.py, line 1044, in _find_and_load
    return _find_and_load_unlocked(name, import_):
  File /data/media/0/src/python3/src/Lib/importlib/_bootstrap.py, line 994, in _find_and_load_unlocked
    _call_with_frames_removed(import_, parent):
  File /data/media/0/src/python3/src/Lib/importlib/_bootstrap.py, line 241, in _call_with_frames_removed
    return f(*args, **kwds):
  File /data/media/0/src/python3/src/Lib/importlib/_bootstrap.py, line 1044, in _find_and_load
    return _find_and_load_unlocked(name, import_):
  File /data/media/0/src/python3/src/Lib/importlib/_bootstrap.py, line 1004, in _find_and_load_unlocked
    parent_spec = parent_module.__spec__:

[RonnyPfannschmidt]

@greyblue9 thansk for the update, so i will update the vendored copy next

[RonnyPfannschmidt]

@greyblue9 i finished the upgrade of the vendored copy, it be neat to get a quick verify just in case so we can prepare a release

[pawamoy]

I copied the contents of https://github.com/pytest-dev/apipkg/blob/main/src/apipkg/__init__.py into ~/.cache/pdm/packages/py-1.10.0-py2.py3-none-any/lib/py/_vendored_packages/apipkg/__init__.py (this is where PDM installs the py package), ran tests again on Python 3.11, and they passed 👍

[RonnyPfannschmidt]

@pawamoy awesome, thanks

[hroncok]

Now when Python 3.11.0a1 is out, we (Fedora) got hit by this on our CI.

[greyblue9]

@greyblue9 i finished the upgrade of the vendored copy, it be neat to get a quick verify just in case so we can prepare a release

Sorry, I didn't see your comment until just now. What should I do to verify?

[RonnyPfannschmidt]

@greyblue9 no longer needed, initial verification already happened, we need to release now

[greyblue9]

@greyblue9 no longer needed, initial verification already happened, we need to release now

thanks @RonnyPfannschmidt , let me know if I can be of any further help

[hroncok]

@RonnyPfannschmidt Is there anything we can do to expedite the release? We are reporting Python 3.11 test failures to various other open source projects and broken pytest makes it harder to reproduce them for their maintainers. Re-installing py from the master branch works. Let me know if you need to check this py change on all Fedora packages to see if it doesn't break anything else, or if there is anything else we can do. Thanks ;)

[RonnyPfannschmidt]

@hroncok thanks for the ping, i completely forgot that #277 was already preparing that over my cold
im preparing to push a tag once hugos pr passes with the updated date

[RonnyPfannschmidt]

@hroncok release is out, happy testing

[hroncok]

[RonnyPfannschmidt]

closing as released