PEP 585 steps 7-8: Implement repr for GenericAlias

[ethanhs]

This is basically a direct port of the Python code, modulo the loop inside was broken out so I didn't have to concat a new tuple to the parameters tuple.

[gvanrossum]

Thanks! There’s something called _UnicodeWriter that I was hoping to use...

[gvanrossum]

This needs a lot more error checking, alas. :-(

Also (as I said) there's a neat internal API (that we are allowed to use here) named _PyUnicodeWriter that should speed this up. Grep for examples if use.

[ethanhs]

I rewrote things with the _PyUnicodeWriter API and it seems much nicer, so thank you for pointing that out!

I also added the test you requested.

[gvanrossum]

This is great. Thanks so much for doing this for me!

[ethanhs]

Happy to help! Thank you for the review :)

I noticed I missed a static (sorry): #3

If I can help with any of the other implementation, let me know. I can probably implement __class_getitem__, __instancecheck__, __subclasscheck__ for the GenericAlias object.

Then the next step would be making it available in Python code I suppose (since ABCs need it)?

[gvanrossum]

If I can help with any of the other implementation, let me know. I can probably implement class_getitem, instancecheck, subclasscheck for the GenericAlias object.

That would be cool -- though I don't think we even need __class_getitem__ -- list[int][int] already fails. I actually worry that this will be a problem in some cases, e.g. this works with Dict' instead of dict, but fails with the builtin dict` in this branch:

D = dict[str, T]
x: D[int]

Then the next step would be making it available in Python code I suppose (since ABCs need it)?

That's easy. I'm planning to add one line to types.py: GenericAlias = type(list[int]) (UPDATE: Done.)

Slightly more work is adding genericity to these classes, in C (follow the example of dict etc.):

• re.Match
• re.Pattern
• io.IO

And these classes that IIUC exist both in C and in Python, all in collections:

• deque
• defaultdict
• OrderedDict
• Counter
• Chainmap

I expect these to go smoothly. What won't be so easy is making type[...] work: it should be equivalent to Type[...], but when I tried this in an early stage it made int[str] possible, so I backed out of it. But we should somehow make it work (maybe a custom __class_getitem__ that fails if the class isn't exactly type).

We also should be able to pickle and unpickle GenericAlias instances (Serhiy reminded me of this). I think this means implementing __reduce__, but it's probably wise to look for examples.

The real test, however, will be making the classes in collections.abc generic. I'm not sure yet how to attempt this -- perhaps we'll need to add a __class_getitem__ method to each of them that creates a GenericAlias instance? Can we do this with a metaclass? (Probably not, tons of code will break if you add a metaclass somewhere.)

[ethanhs]

this works with Dict instead of dict, but fails with the builtin dict in this branch

Hm, perhaps it would be good to consider allowing dict[str, T][str]. If we disallow that, it would be a backwards incompatible change to typing.Dict, no? In addition, I think type checkers would catch things like list[str][str].

re.Match re.Pattern io.IO

I can work on making these generic, should be straightforward.

deque defaultdict OrderedDict Counter Chainmap

I submitted #2 for deque/defaultdict so those are done :) OrderedDict and Counter inherit from dict, so they work as is (I'll make a PR to add tests for this). Chainmap inherits from collections.abc.MutableMapping so once that is subscriptable we get Chainmap for free.

maybe a custom class_getitem that fails if the class isn't exactly type

I can try this out and open a PR if it works. Well sadly this won't work. The first argument is type, which makes sense since type(int) is type.

The real test, however, will be making the classes in collections.abc generic. I'm not sure yet how to attempt this -- perhaps we'll need to add a class_getitem method to each of them that creates a GenericAlias instance?

Yes I think that will be the best way to do it unfortunately.

[gvanrossum]

(Let's keep adding to this thread with progress.)

• Your addition of __class_getitem__ to _io._IOBase triggered a test failure, which I fixed by also adding one to _pyio.IOBase.

Still to do and straightforward:

• Tests for OrderedDict, Counter (they work, can you add the tests?)
• Adding __class_getitem__ to everything in collections.abc
• Add __instancecheck__ and __subclasscheck__ to GenericAlias to raise TypeError (UPDATE)
• Add __class_getitem__ to contextlib.Abstract*ContextManager. (UPDATE)
• Support pickling of list[int] (add a __reduce__ method?) (UPDATE)

Difficult or controversial:

• Fix the bug where deriving from Tuple causes problems (see below)
• Make type[int] work without enabling int[str]
• Allow list[str][str] after all
• Update typing.py (UPDATE: Probably not!)
  • replace re, Match, Pattern
  • replace io, IO, TextIO, BinaryIO (UPDATE: no, these are a separate mess)
  • (most difficult/controversial) replace everything shadowing classes from collections (e.g. Deque) and collections.abc (e.g. Awaitable)
  • Consider whether we can simplify Generic to use the new types.GenericAlias

[gvanrossum]

Update on the issues revealed by the test_typing.py failures. There are actually several unrelated issues.

A. In typing.py, the dunders of a generic or parameterized class are different!

• __origin__ same thing
• __args__ what PEP 585 calls __parameters__
• __parameters__ only parameters that are free (i.e. are typevars)

For example:

>>> import typing
>>> T = typing.TypeVar("T")
>>> D = typing.Dict[str, T]
>>> D.__origin__
<class 'dict'>
>>> D.__args__
(<class 'str'>, ~T)
>>> D.__parameters__
(~T,)
>>>

B. Generic classes as defined in typing.py only allow __class_getitem__ calls with the same number of values as there are in __parameters__.

C. __orig_class__ of an instantiated generic class with parameters (e.g. MyListint) points to the generic class that was instantiated. (Only for pure Python classes, since this is added to the __dict__, and not if there's no __dict__.)

For example:

>>> class C(typing.Generic[T]): pass
... 
>>> x = C[int]()
>>> x.__class__
<class '__main__.C'>
>>> x.__orig_class__
__main__.C[int]
>>> 

This is done by explicitly assinging to obj.__orig_class__ in typing._GenericAlias.__call__. The assignment to __orig_class__ is ignored if it fails (e.g. if class C has __slots__). It would not be very hard to implement this in the C version of GenericAlias. In addition, we might be able to implement Saul Shanabrook's feature request better -- i.e. pass the GenericAlias instance to class methods instead of the origin class.

[gvanrossum]

I'm inclined to rename __parameters__ to __args__, add a new __parameters__ (calculated lazily) that only gives the type vars, and allow things like dict[T, str][str] but not list[int][int], by looking at __parameters__. (To make test_typing.py pass, this would have to do a dynamic lookup of self.__parameters__, since in some cases that value is provided by a base class.)

But I'm inclined not to set __orig_class__ even for instances of pure Python generic classes, because it adds an extra entry to self.__dict__ for each instance created. This seems a bad idea if we want to encourage generics. (Maybe the test in test_typing.py that will be broken by not doing this can be fixed by somehow adding a different __class_getitem__ method to the class created by class C(Tuple[int]): pass.)

[ethanhs]

(Let's keep adding to this thread with progress.)

Works for me!

Your addition of __class_getitem__ to _io._IOBase triggered a test failure, which I fixed by also adding one to _pyio.IOBase.

Ah, sorry, I've just been running the test_genericalias, I'll run the full test suite in future :)

pass the GenericAlias instance to class methods instead of the origin class.

I'm a little concerned a builtin type could special case its behavior based on something like type(cls) == dict or something and having cls be a different type breaking things...

I'm inclined to rename __parameters__ to __args__

Works for me, that is what I called it in my original implementation :)

add a new __parameters__ (calculated lazily) that only gives the type vars, and allow things like dict[T, str][str] but not list[int][int], by looking at __parameters__

This seems reasonable.

But I'm inclined not to set __orig_class__ even for instances of pure Python generic classes, because it adds an extra entry to self.__dict__ for each instance created.

Yeah that seems too high a cost.

(Maybe the test in test_typing.py that will be broken by not doing this can be fixed by somehow adding a different __class_getitem__ method to the class created by class C(Tuple[int]): pass.)

The point of __orig_class__ as I understand it is to point to the subscripted type in an instance, but that isn't needed, is it? Perhaps just removing the test is best? It seems like the best thing (if I am reading what you said correctly) is to not support __orig_class__. And the API is private, so I don't think it would be breaking to remove it. (Though perhaps I am missing something here)

[gvanrossum]

Ah, sorry, I've just been running the test_genericalias, I'll run the full test suite in future :)

No need, it takes forever. And I'm on it.

[ethanhs]

Ok great! Also I'm happy to add tests for OrderedDict and Counter, might start adding __class_getitem__ to collections.abc and contextlib items as well.

[gvanrossum]

Great!

[ethanhs]

Okay so I fixed (I think) the test in #5. I opened #6 for __instancecheck__ and __subclasscheck__ for GenericAlias.

I looked more into type[int] and I think the best option is to write a __class_getitem__ where we check that cls.__name__ == "type". Even in Objects/abstract.c, where __class_getitem__ is implemented, there isn't enough information.

I will probably work on __reduce__ once #6 is merged. For __eq__ I'm not exactly sure what should be done with Unions, perhaps we could modify them to support equality checks and offload to that.

So that's the easy bits :)

This leaves the difficult or controversial items as you said:

Fix the bug where deriving from Tuple causes problems

I think I missed the description of this?

Make type[int] work without enabling int[str]

See above proposal.

Allow list[str][str] after all

You seem to have a plan for this, so I'll focus on other things.

Update typing.py

replace re, Match, Pattern

I think this makes a lot of sense.

replace io, IO, TextIO, BinaryIO (UPDATE: not sure about the latter two, they are almost protocols)

All 3 of them seem to be protocol-like, so perhaps not? I don't have much knowledge in this part.

(most difficult/controversial) replace everything shadowing classes from collections (e.g. Deque) and collections.abc (e.g. Awaitable)

Well I tried the naive way and it didn't work :)
We'd have to figure out a way to limit instantiation as well. Perhaps a subclass of GenericAlias with an overriden __new__ or something. Though you can't make GenericAlias a base hmmm.

Consider whether we can simplify Generic to use the new types.GenericAlias

It looks like we could probably do this.

[gvanrossum]

Make sure you read Lukasz' email to typing-sig. He's okay with several of my proposals (esp. __parameters__ -> __args__) but pushing back on disallowing isinstance/issubclass. I prepared a PR to the PEP, see python/peps#1289 (please review if you have time).

I looked more into type[int] and I think the best option is to write a class_getitem where we check that cls.__name__ == "type". Even in Objects/abstract.c, where __class_getitem__ is implemented, there isn't enough information.

If we're checking that, can't we just check that cls == &PyType_Type? After all we're in C code.

will probably work on __reduce__ once #6 is merged.

Great!

For __eq__ I'm not exactly sure what should be done with Unions, perhaps we could modify them to support equality checks and offload to that.

I think I have a plan, let me do this after I've finished the new __parameters__ thing.

Fix the bug where deriving from Tuple causes problems

I think I missed the description of this?

Scroll up a few comments here. But no worries, I have a fix in my pep585b branch.

All 3 of them [IO, TextIO, BinaryIO] seem to be protocol-like, so perhaps not? I don't have much knowledge in this part.

Heh,. I looked more carefully and these aren't even related to io.IO, despite suggestions that they are. Looking at typeshed, this is still a gigantic mess, so let's not touch any of this.

And the more I think about it, the less I want to touch typing.py... So let's concentrate on the C code and adding generics outside typing.py. (E.g. look for things that are generic in typeshed -- like queue.Queue.)

[ethanhs]

E.g. look for things that are generic in typeshed -- like queue.Queue

Ah yeah I'll look through typeshed for things that are Generic and start adding them to tests (and fixing the ones that aren't!).

[ethanhs]

Okay, I have now grepped through typeshed, here are all of the things that should be generic according to typeshed:

• type, tuple, list, dict, set, frozenset
• collections.defaultdict, collections.deque, collections.OrderedDict, collections.Counter, collections.UserDict, collections.UserList
• re.Pattern, re.Match
• contextlib.AbstractContextManager, contextlib.AbstractAsyncContextManager
• collections.abc.Awaitable, collections.abc.Coroutine, collections.abc.AsyncIterable, collections.abc.AsyncIterator, collections.abc.AsyncGenerator, collections.abc.Generator, collections.abc.Iterable, collections.abc.Iterator, collections.abc.Reversible, collections.abc.Container, collections.abc.Collection, collections.abc.Callable, collections.abc.Set, collections.abc.MutableSet, collections.abc.Mapping, collections.abc.MutableMapping, collections.abc.MappingView, collections.abc.KeysView, collections.abc.ItemsView, collections.abc.ValuesView, collections.abc.Sequence, collections.abc.MutableSequence
• types.MappingProxyType
• types.AsyncGeneratorType
• itertools.chain
• enumerate
• difflib.SequenceMatcher
• filecmp.dircmp
• fileinput.FileInput
• mailbox.MailBox, mailbox.ProxyFile
• weakref.KeyedRef
• _weakref.ReferenceType
• _weakrefset.WeakSet
• contextvars.Token
• dataclasses.Field
• functools._lru_cache_wrapper
• functools.partialmethod, functools.partial, functools.cached_property
• queue.Queue, queue.SimpleQueue
• tempfile.TemporaryDirectory
• ctypes.LibraryLoader
• _ctypes.PyCArrayType (aka ctypes.Array)
• multiprocessing.pool.ApplyResult
• multiprocessing.managers.ValueProxy
• multiprocessing.shared_memory.ShareableList
• multiprocessing.queues.SimpleQueue
• os.DirEntry
• os.PathLike
• unittest.case._AssertRaisesContext
• urllib.parse._NetlocResultMixinBase
• concurrent.futures.Future, concurrent.futures.thread._WorkItem
• http.cookies.Morsel

Need to modify for GenericAlias:

• subprocess.Popen, subprocess.CompletedProcess
• tempfile.SpooledTemporaryFile

Need to make un-generic (oops):

• mmap.mmap
• ipaddress._BaseNetwork

How should we approach tackling these? I can start working on implementing them in smaller batches of PRs. Also for things that are actually private, should we even bother? If it is in typeshed perhaps that indicates people are using it?

[ambv]

I don't think we should enumerate every single generic type (more types will become generic as they are added to Python anyway). Perhaps limiting it to only the items in typing/abcs/etc that would be supplanted would be best?

The PEP says in "forward compatibility" that the expectation is that all new types added to Python will support generic aliasing. The lack of such guarantee is a sure way for typing to always feel hald-baked and not treated seriously by Python core.

In this sense, the more types we cover right now, and list those explicitly, the better.

[gvanrossum]

It’s your PEP, go for it!

[ethanhs]

I personally was thinking that we don't enumerate them all in the PEP, but say "types in the standard library that are generic in typeshed should become generic" it makes the PEP easier to read. I do think we should make all of these types generic.

But as Guido said, it's your PEP :P

[gvanrossum]

"types in the standard library that are generic in typeshed should become generic"

That's not a bad idea at all! I hope Łukasz likes it enough to add some form of this to his PEP.

[gvanrossum]

I just discovered an issue:

from typing import Union
>>> Union[int, list[int]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/guido/pep585/Lib/typing.py", line 262, in inner
    return func(*args, **kwds)
  File "/Users/guido/pep585/Lib/typing.py", line 360, in __getitem__
    parameters = _remove_dups_flatten(parameters)
  File "/Users/guido/pep585/Lib/typing.py", line 234, in _remove_dups_flatten
    all_params = set(params)
TypeError: unhashable type: 'types.GenericAlias'
>>> 

I guess we need to add a tp_hash implementation to the GenericAlias type. I guess it could follow this from typing.py:

    def __hash__(self):
        if self.__origin__ is Union:
            return hash((Union, frozenset(self.__args__)))
        return hash((self.__origin__, self.__args__))

[gvanrossum]

(Except there is no Union type in C. Though PEP 604 might add one.)

[ethanhs]

I don't think we need to worry about Union until it is implemented in C, right? Because unless someone manually does types.GenericAlias(Union, ...) then our __origin__ is never going to be Union.

[gvanrossum]

The problem is that the Union in typing.py (I.e. the old one, in Python) doesn’t accept list[int]. Adding a tp_hash will fix that. Probably the same for Generic[list[int]].

[ethanhs]

Understood, I more meant that we don't have to special case the hash of GenericAlias like in the snippet above if __origin__ is Union. So I think (the C equivalent of)

def __hash__(self):
    return hash((self.__origin__, self.__args__))

should be fine?

[gvanrossum]

Oh. Yes. Anyway, I pushed a tp_hash implementation. A thing I found by testing this: something typing.Union[list[T], int] doesn't find the T inside the list[T] (but it does if you use typing.List[T] instead). This probably also needs to be fixed -- we need to make sure that you can mix PEP 584 generics with constructs that are only available from typing.py. (Another would be class C(list[T]): ....)

[gvanrossum]

Okay, I pushed a fix for Union[list[T], int].

But I don't think it will be easy to make class C(list[T]) work. TBH I feel on unsure footing here, but I believe that the same code that makes class C(list[int]) act like class C(list) -- the __mro_entries__ method, added by PEP 560 -- makes it hard to discover the type variable at runtime. So unless someone has a deep insight I propose to not support that.

@ambv Do you have an opinion here? Is it okay that class C(list[T]) won't work? People will have to use class C(List[T]) instead.

[ethanhs]

btw, @gvanrossum it seems the tests are not being run on your PR to CPython due to a merge conflict. Also I found the Windows build fails, so I opened #16

[gvanrossum]

OK, I did a merge and fixed the conflict. Hopefully this will run the tests?

[gvanrossum]

So I discovered that dir(list) and dir(list[int]) produce entirely different results. It seems clear that the latter introspects the GenericAlias type instead of its __class__ attribute.

I was reminded of this by https://bugs.python.org/issue40098. I tested this using the following script:

def two_up(a, b):
    sa = set(a)
    sb = set(b)
    la = sorted(a)
    lb = sorted(b)
    lab = sorted(set(a) | set(b))
    for key in lab:
        l = str(key) if key in sa else ""
        r = str(key) if key in sb else ""
        print(f"{l:20s}  {r:20s}")

print(f"{'dir(list[int])':20s}  {'dir(list)':20s}")
two_up(dir(list[int]), dir(list))

[ethanhs]

Okay so the PEP has been accepted (congrats Łukasz!) here are the TODOs I see:

• consider a cache like typing.py (it looks like it is essentially an LRU cache of __getitem__, perhaps we can use the C lru_cache implementation?)
• fix dir (could we define __dir__ to pass along dir(list) + (__origin__, __args__, __parameters__)? )
• make all the other types I list above generic.
• document it!

[gvanrossum]

Yeah, that looks like about it. But we can do all those in follow-up PRs -- I don't want to weigh down the initial PR any more than we need to.

Oh, and we'll need docs too (though for the alpha the PEP will suffice).

[gvanrossum]

Hi @ethanhs I mentioned your comment in https://bugs.python.org/issue39481 -- you may want to add yourself to the nosy list there.

[ethanhs]

Hey Guido, thanks! I'll definitely do that. I also will probably start working through the list.