Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix type narrowing of == None and in (None,) conditions #15760

Merged
merged 7 commits into from
Aug 2, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions mypy/checker.py
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,13 @@
is_literal_type,
is_named_instance,
)
from mypy.types_utils import is_optional, remove_optional, store_argument_type, strip_type
from mypy.types_utils import (
can_be_none,
is_optional,
remove_optional,
store_argument_type,
strip_type,
)
from mypy.typetraverser import TypeTraverserVisitor
from mypy.typevars import fill_typevars, fill_typevars_with_any, has_no_typevars
from mypy.util import is_dunder, is_sunder, is_typeshed_file
Expand Down Expand Up @@ -5658,7 +5664,7 @@ def has_no_custom_eq_checks(t: Type) -> bool:
)
if (
collection_item_type is not None
and not is_optional(collection_item_type)
and not can_be_none(collection_item_type)
and not (
isinstance(collection_item_type, Instance)
and collection_item_type.type.fullname == "builtins.object"
Expand Down Expand Up @@ -6065,7 +6071,7 @@ def refine_away_none_in_comparison(
non_optional_types = []
for i in chain_indices:
typ = operand_types[i]
if not is_optional(typ):
if not can_be_none(typ):
non_optional_types.append(typ)

# Make sure we have a mixture of optional and non-optional types.
Expand Down
4 changes: 2 additions & 2 deletions mypy/plugins/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
deserialize_type,
get_proper_type,
)
from mypy.types_utils import is_optional
from mypy.types_utils import can_be_none
from mypy.typevars import fill_typevars
from mypy.util import get_unique_redefinition_name

Expand Down Expand Up @@ -141,7 +141,7 @@ def find_shallow_matching_overload_item(overload: Overloaded, call: CallExpr) ->
break
elif (
arg_none
and not is_optional(arg_type)
and not can_be_none(arg_type)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to me that this logic should also use the new can_be_none(), but I'm not sure how to test this code path.

and not (
isinstance(arg_type, Instance)
and arg_type.type.fullname == "builtins.object"
Expand Down
5 changes: 5 additions & 0 deletions mypy/types_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,11 @@ def is_optional(t: Type) -> bool:
)


def can_be_none(t: Type) -> bool:
t = get_proper_type(t)
return isinstance(t, NoneType) or is_optional(t)


def remove_optional(typ: Type) -> Type:
typ = get_proper_type(typ)
if isinstance(typ, UnionType):
Expand Down
27 changes: 27 additions & 0 deletions test-data/unit/check-narrowing.test
Original file line number Diff line number Diff line change
Expand Up @@ -1261,3 +1261,30 @@ def g() -> None:
def foo(): ...
foo()
[builtins fixtures/dict.pyi]


[case testNarrowingOptionalEqualsNone]
from typing import Optional

class A: ...
Copy link
Member

@sobolevn sobolevn Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about the difference between classes with and without custom __eq__?

In the original issue str instances can never be equal to None.

But, if some type has custom __eq__ - we cannot be sure about this anymore.

Copy link
Contributor Author

@intgr intgr Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is quite open-ended, there are so many nuances here that I'm left guessing what I should do.

This particular bug or fix isn't currently affected by __eq__ method.

Are you asking that I should add tests with str also? And add __eq__ method to A? It miiiight be helpful in the future when narrowing logic changes.

In the original issue str intances can never be equal to None.

One could possibly have a subclass instance of str that overrides __eq__. But is that considered unsound usage? I dunno.


There is another slightly related unsoundness case, when objects consider themselves equal to None. But that would be a separate bug and a harder discussion whether a fix would reduce the usefulness of narrowing.

https://mypy-play.net/?mypy=1.4.1&python=3.11&gist=d6199eb69f69c933ced7da5c51c25a50

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is good enough for me, thanks! :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened another issue for the described case: #15764


val: Optional[A]

if val == None:
reveal_type(val) # N: Revealed type is "Union[__main__.A, None]"
else:
reveal_type(val) # N: Revealed type is "Union[__main__.A, None]"
if val != None:
reveal_type(val) # N: Revealed type is "Union[__main__.A, None]"
else:
reveal_type(val) # N: Revealed type is "Union[__main__.A, None]"

if val in (None,):
reveal_type(val) # N: Revealed type is "Union[__main__.A, None]"
else:
reveal_type(val) # N: Revealed type is "Union[__main__.A, None]"
if val not in (None,):
reveal_type(val) # N: Revealed type is "Union[__main__.A, None]"
else:
reveal_type(val) # N: Revealed type is "Union[__main__.A, None]"
Comment on lines +1273 to +1289
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

meant to fail?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you're asking. This test passes.

Mypy doesn't support type narrowing for these cases yet, I'm only fixing its buggy behavior here in this PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh, got it now! 👌

Copy link
Contributor

@ikonst ikonst Jul 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I assume actually narrowing is much more work, or simply unrelated and could be done incrementally?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's more tricky for sure, and quite rare in Python code. Not as easy as supporting is None, since classes (even a subclas of str) can override __eq__ and return True when compared to None.

I looked into this code because I want to solve #9718, I'd rather put my effort there for now :)

[builtins fixtures/primitives.pyi]
3 changes: 2 additions & 1 deletion test-data/unit/fixtures/primitives.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,8 @@ class memoryview(Sequence[int]):
def __iter__(self) -> Iterator[int]: pass
def __contains__(self, other: object) -> bool: pass
def __getitem__(self, item: int) -> int: pass
class tuple(Generic[T]): pass
class tuple(Generic[T]):
def __contains__(self, other: object) -> bool: pass
class list(Sequence[T]):
def __iter__(self) -> Iterator[T]: pass
def __contains__(self, other: object) -> bool: pass
Expand Down