Repository navigation
Why is "incompatible with supertype" an error? #1237
Description
Activity
It violates the Liskov Substitution Principle. You may want to Google that. In particular the problem is that if we have a Foo instance and we call its factory() method, the typechecker expects that it returns a str; but when that Foo instance is actually a Bar instance, this assumption is incorrect. Note that
isinstance(x, Foo)is still true if x is a Bar instance.Reacted by Tommy Lees, Manuel Wedler, Pavlo Nevmerzhytskyi, Jhon Ramirez, ahallermed, Nipun Sadvilkar, Alexander Rodionov, Ali Nasrullayev, Justin Furuness, Alan Evangelista and 2 moreOn 24 Feb 2016, at 7:32 pm, Guido van Rossum notifications@github.com wrote:
It violates the Liskov Substitution Principle. You may want to Google that. In particular the problem is that if we have a Foo instance and we call its factory() method, the typechecker expects that it returns a str; but when that Foo instance is actually a Bar instance, this assumption is incorrect. Note that isinstance(x, Foo) is still true if x is a Bar instance.
What does that mean regarding the effectivness of mypy on code that exhibits that sort of behaviour? Is it still useful?
My actual use case is in stub files describing a C++ API wrapped in Python. factory() is non-virtual in both Foo and Bar - they are completely unrelated and just happen to have the same name.
Is there a case for treating stub files differently, maybe by providing something like @overload that tells mypy that Bar.factory() is not an override of Foo.factory()?
Phil
You can also use
# type: ignorein the subclass method definition and mypy will stop complaining about it.Reacted by Jonathan Abila and Rick UngPython doesn't have the concept of non-virtual functions, so what you're suggesting is unfortunately unsound.
For example, you could have some function:
def f(x: Foo) -> str: return x.factory()
Because of the way subtyping relationships work in mypy, it's valid to pass a
Baranywhere you need aFoo(see Guido's comment about the Liskov Substitution Principle above). But ifBar.factoryreturns anint, passing aBartofis clearly invalid.At the moment I'm using mypy to validate stub files I am generating. I'm not sure if this error is saying that, in an otherwise correct stub file, mypy can't do its job properly? Or is it saying that the stub file is incorrect no matter what tool is using it?
Phil, does the real code where you hit this happen to use
@staticmethod? Because that could be considered a different case.Alternatively (if you don't like the
# type: ignoresolution) it might be possible to refactor the classes so that Bar is not a subclass of Foo? (What they do have in common could be moved to a base class of Foo.)Regarding whether the stub files are valid or not, I think that a stub file containing the 6 lines of code you showed us should be considered objectively invalid. Whether you can read that in the letter of PEP 484 is a different issue.
Let me try to clarify some more. Suppose we have a file a.py like this:
from defs import Foo def buzz(f: Foo) -> str: return f.factory()And suppose we have b.py:
from defs import Bar from a import buzz b = Bar() buzz(b)There is a type error here, since at run time
buzz(b)will return an int, despite buzz's signature telling us that it returns a str. Where should the type error be reported? Bar is a subclass of Foo, so the call buzz(b) is not at fault, so b.py type-checks (and if the buzz() call is used, it will be treated as returning a string, since that's what a.py says it returns). Also, Foo.factory() returns a str, so a.py is not at fault either. Therefore the only other place that can be faulted is defs.py[i], and that's what you're seeing. I don't think that's unreasonable or unexpected.Perhaps the best solution really is to make it so that Bar is not a subclass of Foo. Is that reasonable?
On 24 Feb 2016, at 10:53 pm, Guido van Rossum notifications@github.com wrote:
Phil, does the real code where you hit this happen to use @staticmethod? Because that could be considered a different case.
In some cases but not all.
Alternatively (if you don't like the # type: ignore solution) it might be possible to refactor the classes so that Bar is not a subclass of Foo? (What they do have in common could be moved to a base class of Foo.)
I have no control over the API I am trying to describe.
I don't like #type because (I'm guessing) other tools (like an IDE using the stub files to create auto-completion lists) would also ignore it.
Regarding whether the stub files are valid or not, I think that a stub file containing the 6 lines of code you showed us should be considered objectively invalid. Whether you can read that in the letter of PEP 484 is a different issue.
The following is from my real stub files...
class QPixmap(QPaintDevice):
def swap(self, other: 'QPixmap') -> None: ...class QBitmap(QPixmap):
def swap(self, other: 'QBitmap') -> None: ......as far I understand PEP 484 that is the correct way to describe the API in a stub file. If you are saying that it is not possible to create a PEP 484-compliant stub file for this API the I would suggest that PEP 484 isn't much use for real applications.
Even if the stub file is PEP 484-compliant, but the API makes it difficult for mypy to do the type checking it wants to, then it should still handle it more gracefully than it does at the moment. The stub files are for PyQt so, as it stands, mypy cannot be used on PyQt applications. I don't see PyQt as being special in this regard, I imagine that there could be many 3rd-party libraries that break these rules, particularly if they wrap C++ libraries.
I have no control over the API I am trying to describe.
Understood. Do you have control over the stub generating tool?
I don't like #type because (I'm guessing) other tools (like an IDE using the stub files to create auto-completion lists) would also ignore it.
Well, PyCharm is sprinting to support all of PEP 484, and their latest (5.1-EAP) even supports the Python 2.7 signature type comments (e.g.
# type: (int, int) -> float) that I added to the PEP recently. I've also talked to authors of both flake8 and pyline and they are also planning to support# type:comments.Of course I don't know what your favorite IDE is or what its support plan is for any of this.
Regarding this example:
class QPixmap(QPaintDevice): def swap(self, other: 'QPixmap') -> None: ... class QBitmap(QPixmap): def swap(self, other: 'QBitmap') -> None: ...
That's a slightly different situation again. Here it seems you have arguments that vary covariantly, which goes against the "Liskov" model that I explained in an earlier comment here. Interestingly, while most languages (and PEP 484*) declare this invalid, in Eiffel this is actually valid (and they insert a runtime check in case a QBitmap down-casted to a QPixmap is called with a QPixmap argument, since the compiler doesn't catch that error).
...I would suggest that PEP 484 isn't much use for real applications.
That came across unnecessarily harsh (and doesn't match my experience at Dropbox, where we are successfully introducing it to what I consider very much a "real application".)
I would like to propose two ways forward -- a short-term way and a long-term approach. In the short term, I think you can solve this by adding a few well-aimed
# type: ignorecomments to your stubs, or perhaps by using Any in some places. (I hesitate to recommend severing the inheritance between QPixmap and QBitmap, since I imagine that would cause more problems than it solves.)In the longer term, I think we can add a directive to PEP 484 to accommodate your use case. For example, if we were to let you write
from typing import covariant_args class QPixmap... # unchangd class QBitmap(QPixmap): @covariant_args def swap(self, other: 'QBitmap') -> None: ...
That should allow you to choose the Eiffel model over the Liskov model for a specific method. Or if you have use cases where the arguments are just entirely different we could come up with some other decorator name.
- While PEP 484 may currently not explicitly describe the exact rules for how subclass methods are allowed to vary compared to the base class method they override, its mentions of variance (and Python's general insistence on the Liskov world-view) make it clear that it sees the world this way. Maybe we could add words to the PEP to clarify this, but I don't the intention is actually unclear (unless we were to try to do a constitutional-lawyer or Bible/Torah expert level attempt to reinterpret an immutable document to changing times :-).
Reacted by 尼亚Side note: as a temporary fix, you can set the retuen type of both
factorymethods toUnion[str, int].Reacted by Alexandre Poitevin, Ryan Morshead and Bora M. Alper27 remaining items
The wording in PEP 483 is poor; what you infer from the example is right.
Feel free to send a PR with updates to PEP 483 (but try not to fix everything at once -- that's a recipe for death in review).
what you infer from the example is right.
But it is not the whole story. The expanded version of that quoted statement is:
IfU1 = Union[t1, ..., tn]andU2 = Union[s1, ..., sn]and everytiis a subtype of at least one ofsjthenU1is a subtype ofU2. For example:class A: ... class B: ... class C(A): ... class D(B): ... class E(A, B): ... x: Union[A, B] y: Union[C, D, E] x = y # OK
Union types behave just as sets
@ilevkivskyi but that only applies if none of y is in x. Then consistency is expanded to include subtypes of elements in x (covariance).
Which means, covariance isn't part of the problem in the case of IntegerOnlyField failing, since it fails the set member comparison: Union[int, float] > Union[float].
Would this be accurate:
def union_compare(u1, u2): u1 = set(u1) u2 = set(u2) if u1 >= u2: return True if u1 < u2: return False for first in u1: first = first or type(first) for second in u2: second = second or type(second) if issubclass(second, first): return True return False print(repr(union_compare([int, float], [float]))) # True print(repr(union_compare([int, float], [bool]))) # True print(repr(union_compare([int, float], [float, int]))) # True print(repr(union_compare([float], [int, float]))) # False print(repr(union_compare([int, float], [None]))) # FalseEdit: superset comparison should include equal set
but that only applies if none of y is in x
"That" (I suppose my extended explanation) applies always, since
Xis a subtype ofXfor everyX(and every type is consistent with itself).Which means, covariance isn't part of the problem in the case of IntegerOnlyField failing
I already explained that the unions are not the problem for that example, but contravariance
of callables in arguments, this simpler example fails as well:class X: ... class Y(X): ... class B: def method(self, arg: X) -> None: ... class C(B): def method(self, arg: Y) -> None: ... # Incompatible with supertype ...
while this works:
class B: def method(self, arg: Y) -> None: ... class C(B): def method(self, arg: X) -> None: ... # OK
I even sent you some links above, but it looks like you didn't read them.
Reacted by WillQI did read them, and totally get the example above - but I don't see how they apply in the case of unions.
Given that:
>>> issubclass(float, int) False >>> issubclass(int, float) FalseWe can rule out any variance rules of the members, or issubclass isn't the correct test.
If issubclass is the correct test, then I do not understand the difference in Union[int] (succeeds) and Union[float] (fails) in comparison with Union[int, float].
If is issubclass is not the correct test and float is a subtype of int, then what is different is that Union[int, float] contains interfaces that are not in implemented in Union[float].And this is the fundamental part I was missing. The initial definition of subtype in PEP483 supports this:
- every value from second_type is also in the set of values of first_type; and
- every function from first_type is also in the set of functions of second_type
But then my inference that subset comparison equals subtype contradicts it:
- Union[int, str] is a subtype of Union[int, float, str]
only works cause everything float implements is already in int, not because int and str are in int, float, str.
If this is correct, then I know how to avoid the confusion in the union section of PEP 483.
Mypy treats
intas a subtype offloat, even thoughintis not a subclass offloat. This is an exception to how things normally work.Thank you all!
Concluding:
- even though,
float.from_bytes()fails and is unsafe, float passes contra-variance test because of special treatment in mypy and is assumed to havefrom_bytesfromint. - Union[int, float] is therefore reduced to Union[float] (less specific)
- specifying unions that contain members with a subtype relationship only adds confusion
- The subtype relationship of unions is not inferred by set membership alone.
Contra-variant behavior of method arguments with Union is best illustrated:
class Field: def to_python(self, value: Union[str, int]): ... class MultiValueField: def to_python(self, value: Union[str, int, List]): ... # OK
Even though it feels natural that the base class should be less specific, one must actually think that the specialized class is capable of handling more.
- even though,
__init__ and __new__are special, mypy allows to override them literary as you wish, no restrictions. (There is a reason for this, otherwise how would you put something in a covariant container, plus this (i.e. incompatible__init__) is very common in practice).Is there any way currently to extend this property to other methods on demand? The repository i'm trying to type has a pattern of
@classmethods calledcreatewhich are used as alternative constructors - these aren't intended to be related to the inheritance hierarchy.At the moment I can bypass via
type: ignore- but I don't really want to ignore them, I just want to tell mypy not to treat these class methods as related.Reacted by Ellis Breenbut I don't really want to ignore them, I just want to tell mypy not to treat these class methods as related.
type: ignoredoesn't ignore types (or anything) it ignores type errors. Essentially, it just silences all errors at the line where it is placed, but has literally no other effects.type: ignoredoesn't ignore types (or anything) it ignores type errors. Essentially, it just silences all errors at the line where it is placed, but has literally no other effects.So if i silence an error at the function definition, i'll still get type errors if I use that function in a way that doesn't respect the definition elsewhere?
i'll still get type errors if I use that function in a way that doesn't respect the definition elsewhere?
Yes.
Reacted by Glen Mailer, Raphael Gaschignard, Kai Willadsen, Jakub Molinski and Ellis Breen- added a commit that references this issue
on Dec 25, 2018 I don't think that there is anything actionable here. It's now possible to use
# type: ignore[override]to specifically silence the incompatible override error (and in a way that is hopefully less confusing than# type: ignore).Reacted by jepperaskdk
Why does mypy consider...
to be an error? While it may be poor design, it isn't wrong.
Thanks,
Phil