Allow using TypedDict for more precise typing of **kwds

There are some situations where a user wants to have more precisely typed **kwds. Current syntax only allows homogeneous **kwds:

def fun(x: int, *, **options: str) -> None:
    ...

However, in situations with many heterogeneous options listing all options in the signature could be verbose and will require rewriting some existing code. There is a vague idea to allow TypedDict for such situations. For example:

class Options(TypedDict):
    timeout: int
    alternative: str
    on_error: Callable[[int], None]
    on_timeout: Callable[[], None]
    ...

def fun(x: int, *, **options: Options) -> None:
    ...

Maybe for such cases the TypedDict used should be automatically understood as defined with total=False. Also it is worth mentioning that this feature will allow reusing the TypedDicts in modules where several functions have same (or similar) option sets.

The proposed syntax is problematic, as it could also mean a homogeneous **options where each value is an Options. We could work around this with some extra syntax, but it could look messy. Here's an idea:

from mypy_extensions import Expand
...
def fun(x: int, *, **options: Expand[Options]) -> None:
    ...

Expand[...] could also be used for variadic type variables in case we add support for them (for example, Callable[[int, Expand[X]], None] if X is a variadic type variable).

I don't like the idea of implicitly assuming total=False.

Yes, the syntax would be ambiguous. Introducing Expand only for this purpose may be not worth it, but using it also for variadic type variables is a good idea (and it feels natural).

I don't like the idea of implicitly assuming total=False.

OK, let's then drop this. In any case it is easy to add it explicitly (I think it will be used often, since keyword arguments are often optional).

Raising priority to normal, since this is a very common request (and we have agreed on using Expand[...]).

@ilevkivskyi

@ilevkivskyi @JukkaL I'm in pretty desperate need of this so I might be able to justify working on this to my employer. Do you have any ideas about how to implement this? If not, are there areas of the code-base that I should look at in order to understand how to approach the problem?

Although this is not super tricky feature, it may be not the best choice for the first contribution, unless you like to live dangerously :-)

You will need to add a new special form Expand[...] to mypy_extensions (where TypedDict lives). This should be allowed only as annotation to **kwars, and must have only one type parameter which is a TypedDict. Then look at all things that contain words formal_to_actual and/or actual_to_formal, they might need to be updated.

Also it may be not the best time to do this, since we are in the middle of a big refactoring of semantic analyzer (one of the first steps in mypy logic, that binds names to symbols). It will take two-three more weeks to finish. Otherwise you would need to repeat some parts of logic twice: in mypy/semanal.py and in mypy/newsemanal/semanal.py.

@ilevkivskyi

@ilevkivskyi, hmm maybe I'll hold off then. Let me know if there's any work a new contributor could provide in order to help out.

@rmorshea

@rmorshea You can try one of these: good-first-issue (unless they are too easy/not interesting).

I got around to hacking something together in #7051. It's obviously very rough but hopefully I can build something off of that.

BTW, I think Expand completely breaks the typesystem. What does this do?

x: Expand[dict[str, int]] = 42

You could argue that this should work because having a type that just works for **kwargs is ugly. You could also argue that this is nonsense and should not be allowed.

I think Expand cannot be called a type. IMO the proper solution would be to introduce a breaking change in the typesystem and make **kwargs: X expand X automatically. Code that looks like this right now:

def foo(**kwargs: int): ...

would have to be changed to:

def foo(**kwargs: Dict[str, int]): ...

If you don't provide bar as a kwarg, it is an error because bar is a required field in the TypedDict.

Ah but we can specify totality like:

class MyKwargs(TypedDict, total=False):
    foo: str
    bar: int

So bar or any key is not required anymore. Does Unpack still report this as an error?

Unpack does not support Union.

I now realize it's not Unpack that should support Union, it should be TypedDict. And TypedDict already does. MyTypedDict | OtherTypedDict means either of the two, not combining them. Maybe what I meant was like Unpack[MyTypedDict, OtherTypedDict] where it unpacks multiple TypedDicts. But I guess that would complicate things.

Thank you!

@krystean

@krystean Just to add to what you said, a small thing I found out yesterday is that you can also use Required and NotRequired for TypedDict if you want to have more control on individual keys (total= is for all keys of the typed dict).

See PEP-655 (also available via typing_extensions).

I am interested in sponsoring work on support for Unpack[T] in mypy if anyone is interested in putting up a PR for this and would find sponsorship helpful.

this should also consider partial views on a class

for example if i had a type like

@dataclass
class MyModel:
   id: uuid
   name: str
   hostname: str
   memory_size: int

and i want to declare helpers like

# bad names
Magic = SubsetTypedDict(MyModel, exclude=["id"], total=False)

def wait_for_update(model_or_id: Model|uuid, **kw: Magic) -> Model:
   ...

If you don't provide bar as a kwarg, it is an error because bar is a required field in the TypedDict.

Unpack does not support Union.

This works for me

from typing import Callable, TypedDict, Union
from typing_extensions import NotRequired, TypedDict

from typing_extensions import Unpack


class MyFuncKwargs(TypedDict):
    a: NotRequired[str]
    b: NotRequired[int]


def pint(b: int) -> None:
    print(b)


def pstr(a: str) -> None:
    print(a)

def my_func(f:Union[Callable[[str], None], Callable[[int], None]], **kwargs: Unpack[MyFuncKwargs]) -> None:
    print(kwargs)
    return


my_func(pstr, a="a")
my_func(pint, b=2)
my_func(pint, c=2)
my_func(pint, a=2)

/home/devmessias/phd/pkgs/pyastrx-proj/pyastrx-my/pyright.py
  /home/devmessias/phd/pkgs/pyastrx-proj/pyastrx-my/pyright.py:26:15 - error: No parameter named "c" (reportGeneralTypeIssues)
  /home/devmessias/phd/pkgs/pyastrx-proj/pyastrx-my/pyright.py:27:17 - error: Argument of type "Literal[2]" cannot be assigned to parameter "a" of type "str" in function "my_func"
    "Literal[2]" is incompatible with "str" (reportGeneralTypeIssues)
2 errors, 0 warnings, 0 informations 
Completed in 0.877sec

Are there any prospects to bring Unpack to mypy?

I am interested in sponsoring work on support for Unpack[T] in mypy if anyone is interested in putting up a PR for this and would find sponsorship helpful.

Would be great to have unpack available in mypy

@RonnyPfannschmidt

this should also consider partial views on a class

for example if i had a type like

@dataclass
class MyModel:
   id: uuid
   name: str
   hostname: str
   memory_size: int

and i want to declare helpers like

# bad names
Magic = SubsetTypedDict(MyModel, exclude=["id"], total=False)

def wait_for_update(model_or_id: Model|uuid, **kw: Magic) -> Model:
   ...

@RonnyPfannschmidt Could you make this into an independent issue for this so it gets attention?

dataclass can be unpacked to act as a TypedDict
exclude fields
Being able to pass via **kw: Magic (conformance with Allow using TypedDict for more precise typing of **kwds #4441

P.S. if I understand correctly, you want to be able to take dictionary data,, and hydrate models / dataclasses via **kw. Is that an accurate depiction of what you're getting at?

@tony

@tony i don't want to hydrate, i want to correctly type helper functions for querying, waiting and other actions that replicate all if not most fields of types

It seems Unpack from typing_extensions does more than what PEP 646 originally proposed (i.e. originally only for *args/TypeVarTuple, not also **kwargs/TypedDict). It now seems to also do what Expand was proposed for in PEP 589. This is perfectly fine by me (I love having Unpack), but I have some questions:

Is Expand in PEP 646 also going to be implemented?
Will PEP 646 be updated to indicate that Unpack may also be used for variadic TypeDicts?

I think ideally there should only be 1 name and it should work for both types. (It would be confusing, for instance, to have both Expand and Unpack).

ASIDE:

If it is agreed there should be one name, my vote is for Unpack, simply because it's already implemented and the terminology matches Python terminology (we speak of "packing"/"unpacking" variables, rather than "collapsing"/"expanding" variables).
PEP 646 appears to have implemented a grammar change to utilize starred expressions in indeces beginning in Python 3.11. Could this be expanded to also implement double-starred expressions? There've been discussions here that implementing a ** syntax would require a grammar change. I'm wondering if expanding the grammar change for * would make this less work? This would be a nice-to-have (much like how the grammar was updated to support built-ins in the syntax directly rather than requiring from typing import UppercaseBuiltin and pipes for Union).

There is a new PEP 692 underway for this functionality. It proposes to add ** syntax and use Unpack for backward compatibility. The new PEP is under review, so if you want to comment on it, I'm sure the feedback would be welcome.

Pyright has provisional support for PEP 692 if you want to play with it.

A few additional notes...
PEP 589 didn't specify Expand. The only reference to Expand is in the rejected alternatives section. There are currently no plans to introduce Expand to the typing module.

You asked whether PEP 646 would be updated to cover new usage of Unpack. That's not how PEPs work. They are intended to be point-in-time proposals, not live documentation of new functionality, so it wouldn't be appropriate to update PEP 646 to cover new uses of Unpack. That will be covered in PEP 692.

There is a new python/peps#2620 underway for this functionality. It proposes to add ** syntax and use Unpack for backward compatibility. The new PEP is under review, so if you want to comment on it, I'm sure the feedback would be welcome.

Pyright has provisional support for PEP 692 if you want to play with it.

Hurray! 🎉

I'll be sure to add some comments (would love to see dual support of Unpack for TypeVarTuple and TypedDict).

PEP 589 didn't specify Expand. The only reference to Expand is in the rejected alternatives section. There are currently no plans to introduce Expand to the typing module.

Yes, thank you for clarifying it was rejected (good catch). Thank you for the clarification!

You asked whether PEP 646 would be updated to cover new usage of Unpack. That's not how PEPs work. They are intended to be point-in-time proposals, not live documentation of new functionality, so it wouldn't be appropriate to update PEP 646 to cover new uses of Unpack. That will be covered in PEP 692.

Yes, another good catch. I wasn't aware of PEP 692, which is why I was asking. However, I've seen "update" notes added to PEPs before. For example, PEP 484 has several:

If type hinting proves useful in general, a syntax for typing variables may be provided in a future Python version. (UPDATE: This syntax was added in Python 3.6 through PEP 526.)

Could an update note be added to PEP 484 if and when PEP 692 is approved? This would help readers and add traceability between the two. Something as simple as

UPDATE: Usage of Unpack also for **kwargs and TypedDict is proposed in PEP 692.

would be great.

Allow using TypedDict for more precise typing of **kwds (#13471)

Since kwargs values can have defaults. Unpacking with defaults would be nice to support.
Is it not possible to Unpack a pydantic(BaseModel) ?
This would make integrations with libraries like strawberry easy.

For example

class AllFilters(BaseModel):
    beg_age: int = 21
    end_age: int = 55
    beg_height: int = 120
    end_height: int = 240
    search_term: str = ' '
    gender: str = 'female'

@strawberry.type
class Query:
    @strawberry.field
    async def get_filtered_users(self, **kwargs: Unpack[AllFilters]) -> List[User]:
        '''This function gets all users that match the filter criteria
        Strong default values are set to prevent empty results 
        '''

Navigation Menu

Repository navigation

Allow using TypedDict for more precise typing of **kwds #4441

25 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Allow using TypedDict for more precise typing of **kwds #4441

Description

Activity

JukkaL commented on Jan 9, 2018

ilevkivskyi commented on Jan 9, 2018

ilevkivskyi commented on Mar 15, 2019

rmorshea commented on Mar 15, 2019

ilevkivskyi commented on Mar 16, 2019

rmorshea commented on Mar 18, 2019

ilevkivskyi commented on Mar 18, 2019

untitaker commented on Jun 24, 2019

25 remaining items

krystean commented on Mar 27, 2022

dgellow commented on Apr 6, 2022

rattrayalex commented on Apr 6, 2022

RonnyPfannschmidt commented on Apr 27, 2022

devmessias commented on May 29, 2022

devmessias commented on May 29, 2022

tony commented on May 30, 2022

RonnyPfannschmidt commented on May 30, 2022

adam-grant-hendry commented on Jun 13, 2022

erictraut commented on Jun 13, 2022

adam-grant-hendry commented on Jun 14, 2022

XChikuX commented on Jun 9, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions