Paramspec support in Generic

[shughes-uk]

Trying to define a generic using the backporked paramspec in python 3.

from typing import Generic
from typing_extensions import ParamSpec

p = ParamSpec("p")
class X(Generic[p]):
    pass

You'll run into an exception thrown by this snippet in the Generic class

            if not all(isinstance(p, TypeVar) for p in params):
                raise TypeError(
                    f"Parameters to {cls.__name__}[...] must all be type variables"
                )

In 3.10 this has been changed to allow it as defined by PEP 612

https://github.com/python/cpython/blob/3.10/Lib/typing.py#L1260

From the limitations note in the readme I would expect it to work for static typing checking but fail silently at runtime, instead I get an exception that interrupts program flow.

[JelleZijlstra]

Good catch, I can confirm that this doesn't work. It's also not covered in the CPython test suite (https://github.com/python/cpython/blob/main/Lib/test/test_typing.py), which is why we didn't catch it when backporting to typing_extensions.

This isn't straightforward to fix, because typing.TypeVar doesn't allow subclasses, so we can't make ParamSpec inherit from TypeVar. However, if we add a line __class__ = typing.TypeVar to the definition of typing_extensions.ParamSpec, that's enough to fool isinstance. It's not enough to make Generic work, though: I get AttributeError: 'ParamSpec' object has no attribute '_get_type_vars'. But that should also be fixable. I can spend some time in the next few days trying to get this to work, unless you or someone else reading this is interested in picking it up.

cc @Fidget-Spinner who implemented ParamSpec

[shughes-uk]

Could we resolve by messing with __instancecheck__?

[JelleZijlstra]

@shughes-uk I don't think so, because that would have to be on TypeVar, and we can't change TypeVar in older Python versions.

[TeamSpen210]

Couldn't you just reassign __instancecheck__, or even its __code__ attr?

[JelleZijlstra]

I suppose we could, but that seems more evil than the __class__ hack I suggested.

[gvanrossum]

@Fidget-Spinner

[shughes-uk]

Any ideas on a hacky workaround I could use in the mean time? My use case is class decorators ala

task_params = ParamSpec("task_params")
task_return = TypeVar("task_return")

class SomeDecorator(Generic[task_params, task_return]):
    def __init__(self, func: Callable[task_params, task_return]):
        self.func = func
    def __call__(self, *args: task_params.args, **kwargs: task_params.kwargs) -> task_return:
        return self.func(*args, **kwargs)

[JelleZijlstra]

Monkeypatching TypeVar.__instancecheck__ (as @TeamSpen210 suggested above) could work locally. I don't want to do that in typing_extensions though.

[shughes-uk]

Hmm so i'd have to monkeypatch TypeVars metaclass for It to work I think, which would mean i'm monkeypatching type. Python wont actually let me patch a builtin like type thankfully.

[JelleZijlstra]

In [118]: t = type(TypeVar)

In [119]: t.__instancecheck__ = lambda *args: True

In [120]: isinstance(1, TypeVar)
Out[120]: True

No guarantees that anything else will work as expected after this.

[shughes-uk]

Interesting in python 3.8.10 I get TypeError: can't set attributes of built-in/extension type 'type' when trying that

[shughes-uk]

I managed to pull some shenanigans with

def hacky_hack(obj, typ):
    if typ == TypeVar and isinstance(obj, ParamSpec):
        return True
    else:
        return isinstance(obj, typ)


typing.isinstance = hacky_hack

[Fidget-Spinner]

Good catch, I can confirm that this doesn't work. It's also not covered in the CPython test suite (https://github.com/python/cpython/blob/main/Lib/test/test_typing.py), which is why we didn't catch it when backporting to typing_extensions.

I think in CPython the test is test_user_generics in test_typing. Though it's intentionally omitted for typing_extensions.

https://github.com/python/cpython/blob/main/Lib/test/test_typing.py#L4314

This isn't straightforward to fix, because typing.TypeVar doesn't allow subclasses, so we can't make ParamSpec inherit from TypeVar. However, if we add a line __class__ = typing.TypeVar to the definition of typing_extensions.ParamSpec, that's enough to fool isinstance [...]

I remember trying the __class__ approach some time back to trick the type checks in Generic. It worked well but the only problem was that isinstance(P, TypeVar) would be true for typing_extensions but false for typing since ParamSpec is not a subclass of TypeVar as you already mentioned.

I don't like the asymmetry there as it would mean porting code from typing_extensions to typing isn't as simple as just changing the import for ParamSpec especially if there's runtime introspection involved. However between that and the feature just not existing at all, I'm in favour of tricking Generic. We should probably add a note in the docs saying not to rely on isinstance(P, TypeVar) and use the more universal isinstance(P, (TypeVar, ParamSpec)) after fixing this.

[Fidget-Spinner]

Interesting in python 3.8.10 I get TypeError: can't set attributes of built-in/extension type 'type' when trying that

Woops, I hope that isn't a regression in 3.10-11, it might be :( https://bugs.python.org/issue43908

[JelleZijlstra]

I think in CPython the test is test_user_generics in test_typing. Though it's intentionally omitted for typing_extensions.

https://github.com/python/cpython/blob/main/Lib/test/test_typing.py#L4314

You're right, I missed that test.

I remember trying the __class__ approach some time back to trick the type checks in Generic. It worked well but the only problem was that isinstance(P, TypeVar) would be true for typing_extensions but false for typing since ParamSpec is not a subclass of TypeVar as you already mentioned.

I don't like the asymmetry there as it would mean porting code from typing_extensions to typing isn't as simple as just changing the import for ParamSpec especially if there's runtime introspection involved. However between that and the feature just not existing at all, I'm in favour of tricking Generic. We should probably add a note in the docs saying not to rely on isinstance(P, TypeVar) and use the more universal isinstance(P, (TypeVar, ParamSpec)) after fixing this.

This would only cause problems for people doing manual introspection of TypeVar and ParamSpec though.Use in type annotations like in @shughes-uk's example would just work for users, and that's the primary purpose of typing_extensions. I agree with adding documentation about not using isinstance(..., TypeVar); that can go in the CPython docs for TypeVar.

[Fidget-Spinner]

Interesting in python 3.8.10 I get TypeError: can't set attributes of built-in/extension type 'type' when trying that

I also get a TypeError in 3.11 so the __class__ approach seems more feasible. I have created a PR at #817. Jelle I hope I didn't cause you any double work (sorry if I did).