Proposal: signature copying for kwargs.

There's a quite common pattern in python code which is:

def function(foo, *args, **kwargs):
    # do something with foo
    other_function(*args, **kwargs)
    # possibly do something else

def other_function(color: str=..., temperature: float=..., style: Stylesheet=..., timeout: Optional[int]=..., database_adaptor: Adaptor=..., strict: bool=..., output: IO[str], allow_frogs: bool=..., mode: SomeEnum=...):
    # do something with a lot of options

(a usual subcase of this one is when other_function is actually super().function ). This presents two problems for a static analyzer:

the call from function to other_function can not be type-checked properly because of the *args, **kwargs in the call arguments.
there is no sensible way to annotate function, so calls to it are unchecked.

The problem for me also affects readability of the code (which is for me one of the main problems that annotations tries to address). James Powell from numfocus even gave a pydata talk about the difficulties it brings at https://www.youtube.com/watch?v=MQMbnhSthZQ

Even if theoretically the args/kwargs packing feature of python can be used with more or less arbitrary data, IMO this use-case is common enough to warrant some special treatment. I was thinking on a way to flag this usage, for example:

@delegate_args(other_function)
def function(foo, *args, **kwargs):
    other_function(*args, **kwargs)

This could hint an analyzer so:

On calls to function, the "extra" arguments are checked to match the signature of other_function
The call to other_function is considered valid given that it uses the same arguments (I know that the code above could have modified the content of kwargs, but it's still more checking than what we have now).

For me, even without static analyzer, the readability benefits of seeing

@delegate_args(matplotlib.pyplot.plot)
def plot_valuation(ticker_symbol: str, start: date, end: date, *args, **kwargs): ...

and knowing that plot_valuation accepts any valid arguments from matplotlib's plot function, is worth it.

Gotta make this quick: Yes, we experience this in our own (Dropbox) code bases a lot too. And I agree this is useful for readability even without static checking. I think there are some details to be worked out -- what I often see is that the wrapping function adds some fixed args to the function in delegates too, so those should not be accepted:

def raw(name='me', age=42): ...

def cooked(first_name, last_name, **extra):
    raw(name=first_name + ' ' + last_name, **extra)

cooked(age=100)  # OK
cooked(name='Guido')  # Error

Also of course the wrapper may also have some of its own arguments that are unrelated to the wrapped function.

I've seen this pattern quite frequently as well, and it would be nice to support it somehow. Refactoring existing code to use explicitly spelled out individual arguments instead of *args and **kwargs can be lot of work and make the code harder to maintain, especially if there are many args to propagate.

Expanding on the decorator idea, maybe it should be possible to declare that a function accepts all positional and keyword args another function accepts, except for a set of args. Example:

def raw(name='me', age=42): ...

@delegate_args(raw, except=['name'])
def cooked(first_name, last_name, **extra):
    raw(name=first_name + ' ' + last_name, **extra)

cooked(age=100)  # OK
cooked(name='Guido')  # Error

Implementing this in a type checker looks a little tricky, especially when using both *args and **kwargs. Special-casing the whole thing looks possible, but sometimes programmers mutate the **kwargs dictionary and type checking that would likely require some sort of support for "dict-as-struct".

@JukkaL

OK, to support a bit of this with data I did a small experiment: take a random sample of stdlib files until I got about 100 function definitions with open keyword args; then another random sample in django with 100 more defs, and check manually what they do with the arguments. My finding say that:

17% of the defs with **k are for functions that never use k
18% of the defs with **k actually use k as a dictionary (calling get(), pop(), iterating on it, etc., but also storing it into a variable or attribute, or passing k as a parameter but without **)
40% of the defs with **k pass it as is to another function, without adding more positional or keyword arguments on the call (This percentage also includes the functions that pass *varargs as is too).
24% of the defs with **k pass it (with ** again) to another function, but the call includes some extra arguments (positional or keyword).
1% of the defs with **k use it for self.__dict__.update(k).

Case 1 is trivial and works well already with Any or object. Case 2 is the hard one (when the dictionary is heterogeneous, which appears to be the most frequent scenario, although I didn't collect that data). Case 5 could be a verification target for a type checker but seems quite unusual. My proposal would cover case 3, and with @JukkaL addition, a part of case 4.

So "hard" cases would be reduced to about 20% of the def statements with **kwargs according to this semi-scientific experiment :) .

From what I saw of case 4, actually I think that a lot of cases could be covered automatically. The example that @JukkaL posted is not very common. The common case for those (at least in my sample) are the following two:

def raw(name=..., age=...): ...

@delegate_args(raw)
def cooked(name, **extra):
    raw(name=name.upper(), **extra)
    # here the typechecker could deduce that `**extra` never contains a `name`, because it's a named argument of "cooked"

def raw2(a, b, c, name=..., age=...): ...

@delegate_args(raw)
def cooked2(a, b, c, **extra):
    raw(a+1, b, c, **extra)
    # here the typechecker could deduce that `**extra` never contains `a`, `b`, or `c`, because they are named arguments of `cooked2`

So the except=... proposal is helpful, but in just a handful of cases (which I prefer, and also avoids the "except" keyword). There's also this case (which I included in my group 2) that could benefit from a similar proposal which is:

@delegate_args(raw, include=['foo'])
def cooked(**extra):
    foo = extra.pop('foo', SOME_DEFAULT)
    do_something_with(foo)
    raw( **extra)

I would have declared foo as an explicit argument and perhaps that's what we should recommend, but I'm a bit surprised on how relatively common this pattern is.

@dmoisset

@dmoisset Thanks for the careful analysis! Having data like this makes decisions much easier. So the conclusion seems to be that this would be useful even without support for dict-as-struct. It might be useful for us to run a similar analysis against Dropbox code to get another data point.

I came across this issue, and wanted to present my solution to the problem:

import forge

def raw(name='he', age=42):
    return f'{name} is {age}'

@forge.compose(
    forge.copy(raw, exclude=('name')),
    forge.insert((forge.arg('first_name'), forge.arg('last_name')), index=0),
)
def cooked(first_name, last_name, **extras):
    extras['name'] = first_name + ' ' + last_name
    return forge.callwith(raw, extras)

(and some tests that validate that it works as expected)

import inspect
assert repr(inspect.signature(cooked)) == '<Signature (first_name, last_name, age=42)>'
assert raw(age=100)  == 'he is 100'
assert raw('Guido', age=42) == 'Guido is 42'
assert cooked('Guido', 'VR', 42) == 'Guido VR is 42'

try:
    cooked(name='Guido')  # Error
except TypeError as exc:
    assert exc.args[0] == "cooked() missing a required argument: 'first_name'"

forge is a package I recently released that allows for signature revision (add, remove, modify parameters as well as convert or validate argument values, etc.).

I'm curious as to whether a mypy plugin could be written that would allow for static analysis.

This is quite the mouthful:

@forge.compose(
    forge.copy(raw, exclude=('name')),
    forge.insert((forge.arg('first_name'), forge.arg('last_name')), index=0),
)
def cooked(first_name, last_name, **extras):

How about instead forge.copy just doesn't interfere with arguments in the function it's decorating that don't have * or **? In other words the forge.insert could be automatic. If somehow that turns out to be a problem in some cases it could be turned off with a keyword argument.

I don't understand the need for this:

    extras['name'] = first_name + ' ' + last_name
    return forge.callwith(raw, extras)

Why not raw(name=first_name + ' ' + last_name, **extra)?

So forge can do more than solve the problem discussed here. You bring up an interesting point though, is that a common-enough problem to which a canned decorator should be provided? I'll share a snippet in a little bit.

The reason for callwith is that reconstituting arguments (especially the ordering of positional or keyword parameters is a non-trivial problem:

import forge

def func(a, b, c, d=4, e=5, f=6, *args):
    return (a, b, c, d, e, f, args)

@forge.sign(
    forge.arg('a', default=1),
    forge.arg('b', default=2),
    forge.arg('c', default=3),
    *forge.args,
)
def func2(*args, **kwargs):
    return forge.callwith(func, kwargs, args)

assert forge.repr_callable(func2) == 'func2(a=1, b=2, c=3, *args)'
assert func2(10, 20, 30, 'a', 'b', 'c') == (10, 20, 30, 4, 5, 6, ('a', 'b', 'c'))

The alternative to that is manual interpolation of argument values which is just as big of a problem:

import forge

def func(a, b, c, d=4, e=5, f=6, *args):
    return (a, b, c, d, e, f, args)

@forge.sign(
    forge.arg('a', default=1),
    forge.arg('b', default=2),
    forge.arg('c', default=3),
    *forge.args,
)
def func2(*args, **kwargs):
    return func(
        kwargs['a'],
        kwargs['b'],
        kwargs['c'],
        4,
        5,
        6,
        *args,
    )

assert forge.repr_callable(func2) == 'func2(a=1, b=2, c=3, *args)'
assert func2(10, 20, 30, 'a', 'b', 'c') == (10, 20, 30, 4, 5, 6, ('a', 'b', 'c'))

I explain that further, in the docs.

OK, that's cool, but:

Anyone who writes

def func(a, b, c, d=4, e=5, f=6, *args):

instead of

def func(a, b, c, *args, d=4, e=5, f=6):

deserves to be slapped, and

If I understand correctly, the cooked/raw example can work fine without callwith?

Both are correct :)

Alright, I got a minute to draft out an extend revision:

import functools
import inspect

import forge


class extend(forge.Revision):
    """
    Extends a function's signature...
    """
    def __init__(self, callable, *, include=None, exclude=None):
        # pylint: disable=W0622, redefined-builtin
        self.callable = callable
        self.include = include
        self.exclude = exclude

    def revise(self, previous):
        extensions = forge.fsignature(self.callable)
        if self.include:
            extensions = list(forge.findparam(extensions, self.include))
        elif self.exclude:
            extensions = [
                param for param in extensions
                if param not in forge.findparam(extensions, self.exclude)
            ]

        params = [
            param for param in previous
            if param.kind is not forge.FParameter.VAR_KEYWORD
        ] + list(extensions)

        return forge.FSignature(params)

Usage is straightforward:

def raw(name='he', age=42):
    return f'{name} is {age}'

@extend(raw, exclude=('name'))
def cooked(first_name, last_name, **extras):
    return raw(name=f'{first_name} {last_name}', **extras)

And the tests still pass:

assert repr(inspect.signature(cooked)) == '<Signature (first_name, last_name, age=42)>'
assert raw(age=100)  == 'he is 100'
assert raw('Guido', age=42) == 'Guido is 42'
assert cooked('Guido', 'VR', 42) == 'Guido VR is 42'

try:
    cooked(name='Guido')  # Error
except TypeError as exc:
    assert exc.args[0] == "cooked() missing a required argument: 'first_name'"

The caveats to this approach are that a user must remain wary of the ordering of "parameter kind" and parameters with default values (or use within a @forge.compose(forge.extend(...), forge.sort()) construct.

Add basics of delegate decorator: see python/typing#270

@alexmojaki

python/mypy#5559 has an implementation of the basic proposal with exclude= by @alexmojaki as a mypy extension. Before moving on, I'd like to discuss the design a bit more.

In particular, there's the question of what syntax to use. All examples above use a decorator, but @ilevkivskyi also suggested (in python/mypy#5559) using a TypedDict (another mypy extension) or a protocol-based callable as an annotation. Here's an example using a callable annotation from the above PR:

class DoStuff(Protocol):
    def __call__(a: int, b: int = ..., c: int = ...) -> None: ...

f: DoStuff
def f(*args, **kwargs):
    ...

g: DoStuff
def g(*args, **kwargs):
    f(*args, **kwargs)

This is not optimal if the original function (such as f above) doesn't use *args and/or **kwargs for all arguments, since then we'd need to duplicate the argument names and types, even though that's what the proposal is trying to avoid. So there would arguably only be a benefit if there are two or more functions that delegate to f.

Here's another idea (which will likely be harder to implement):

def f(x: int, y: str = '') -> None:
    ...

g: TypeOf[f]
def g(*args, **kwargs):
    ...

We'd introduce a TypeOf[x] type operator that is equivalent to the type of the expression within square brackets. It could potentially also be used for other things, such as callbacks, though these use cases may be pretty marginal:

def default(x: int, y: str = '') -> None:
    ...

def do_stuff(cb: TypeOf[default] = default) -> int:
    ...

This would only cover uses cases where the signatures of the two functions are identical.

Even if we continue with the decorator proposal, we don't have an agreement on what we'd call it. The ideas above aren't quite self-explanatory enough, in my opinion. The feature is not very widely useful, so I feel like we should try to make name very clear and explicit. Here are a bunch of random ideas (using a real example from subprocess):

@copy_signature(Popen)
def call(*popenargs, timeout: Optional[float] = None, **kwargs)): ...

@inherit_signature(Popen)
def call(*popenargs, timeout: Optional[float] = None, **kwargs)): ...

@with_signature(Popen)
def call(*popenargs, timeout: Optional[float] = None, **kwargs)): ...

@use_signature(Popen)
def call(*popenargs, timeout: Optional[float] = None, **kwargs)): ...

@use_signature_from(Popen)
def call(*popenargs, timeout: Optional[float] = None, **kwargs)): ...

@apply_signature(Popen)
def call(*popenargs, timeout: Optional[float] = None, **kwargs)): ...

Here my rationale is that the decorator actually does no delegation -- the assumption is that the function delegates to another function, but that's actually not enforced by the feature, I assume. So the effect is to take the signature of another function and apply it to the decorated function.

Here are few more random ideas/comments:

Instead of TypeOf[f], that may give an impression that this applies to any other object, I would propose to allow single argument Callable like this:

def f(x: int, y: str = '') -> None:
    ...
g: Callable[f]
def g(*args, **kwargs):
    ...
def do_stuff(cb: Callable[f] = f) -> int:
    ...

...protocol-based callable as an annotation...
So there would arguably only be a benefit if there are two or more functions that delegate to f.

Yes, the same also applies to using TypedDict to annotate **kwargs. Allowing TypedDict in annotations has its own use case, Allow using TypedDict for more precise typing of **kwds mypy#4441, so probably it is a bit orthogonal here.
If we go with the decorator approach, maybe again we can use (now stalled) @declared_type? (I am not sure I like this however.)

My preference would be probably to go with g: Callable[f].

That's great for copying a signature, but it does nothing for the just-as, if-not-more common case where you want to mutate the parameters: #270 (comment)

A decorator allows performing actions at runtime such as setting the __signature__ attribute. This cannot be done with an annotation. To me this is a complete dealbreaker for using an annotation. I could elaborate some reasons for this, but do I need to? Does anyone feel that runtime signatures aren't such an important feature for this proposal?

Does PEP 612 offer a solution to the common usecase of copying the signature of the superclass method when overriding?

I've read through it and while it seems like the use-case of signature-preserving decorators is now well-supported, I can't see any way to indicate 'same as parent' or even 'same as x' where x is another callable. Could anyone tell me if I've just missed something glaringly obvious? I found some parts of the PEP a little hard to digest without first seeing more examples of its use out in the wild.

Honestly, I really like the use of attribute-access syntax to separate a ParamSpec out into P.args and P.kwargs.

If there is not yet a way to meet this need with PEP 612 I would like to start a new thread suggesting the following (in the spirit of PEP 612):

def raw(some: str, /, signature: int, goes: bool, *, here: dict) -> str:
    ...


def cooked(*args: raw.args, **kwargs: raw.kwargs) -> raw.ret:
    ret = raw(*args, **kwargs)
    ...  # do something with ret before returning it

Allowing func.args, func.kwargs, and (possibly, and subject to a name-change) func.ret as special type hints that are valid for any function.

And for the specific use-case of method overriding:

from typing import Super


class Parent:
    def some_method(self, some: str, /, signature: int, goes: bool, *, here: dict) -> str:
        ...


class Child(Parent):
    def some_method(self, *args: Super.args, **kwargs: Super.kwargs) -> Super.ret:
        ret = super().some_method(*args, **kwargs)
        ...  # do something with ret before returning it

If typing.Super were a special object that signified to type-checkers and IDEs that the parent's signature for that method should be used.

So yeah. Are these use-cases covered? If not, has there been a successor thread to this where I could post this as a suggestion? And if not, would it be okay if I started one?

Cheers

@matthewgdv

@matthewgdv if all you want to do is copy the signature, this comment describes a way to do that right now: #270 (comment)

That doesn't really help you though if you want to actually add arguments (e.g.(new_arg, *args, **kwargs)). In that case, I don't really see an obvious way to use ParamSpec to accomplish that.

Perhaps in a future revision the following could be made possible?

P1 = ParamSpec("P1")
P2 = ParamSpec("P2")

def concatenate_functions(f1: P1, f2: P2) -> Concatenate[P1, P2]:
    ...

def f(x: int) -> int: ...
def g(y: int) -> int: ...

h: Callable[[int, int], int] = concatenate_functions(f, g)

There might be a separate issue to track this though - a link here would be great if anyone knows of such an issue.

The Problem

As mentioned above, the most-used cases of **kwargs were passing it to the other function, with or without any additional arguments. However, the new PEP-612's ParamSpec does not cover that case as it does not provide syntax for merging keyword parameters from different functions defined at different places.

Example:

class A:
    def __init__(self, *, param_1: int, param_2: str, param_3: bool = False):
        pass

class B:
    a: A
    
    def create_a(self, *, log: bool = False, **kwargs):
        self.a = A(**kwargs)
        if (log): print("Log!")

Here, B.create_a()'s real signature is create_a(self, *, log: bool = False, param_1: int, param_2: str, param_3: bool = False).
I managed to create a simple decorator that merges signatures and overrides the attribute __signature__, but as this happens at runtime exclusively, neither PyCharm nor mypy are able to handle it correctly.
Since it is one of the most-common usages, as well as MyPy eating default values and still (Jun 2022) not supporting dataclasses, I consider this request as not resolved.

If I am somewhere wrong, please, correct me.

P.S.

If anybody is interested, I am attaching my implementation of merge_signature.
Usage:

from .merge_sig import merge_signature
class B:
    a: A
    
    @merge_signature([ A.__init__ ])
    def create_a(self, *, log: bool = False, **kwargs):
        ...

file:/home/peter/projects/test/merge-sig.tar.gz

I tried different tools for generating stub files, and neither of them succeeded in generating the correct signature
(or even just respecting dataclass).

To your last point, I think there's more than just signature copying. As mentioned in a previous comment it would also be useful if one could add fixed arguments to test that are not a part of f:
def f(x: bool, *extra: int) -> str: ...

@copy_signature(f)
def test(fixed_argument: str, *args, **kwargs): ....

# desired
reveal_type(test)  # Revealed type is 'def (fixed_argument: str, x: bool, *extra: int) -> str'
# actual
reveal_type(test)  # Revealed type is 'def (x: bool, *extra: int) -> str'

I apologize in advance if this has already been solved. I wanted to share what I found with Concatenate and ParamSpec since it took me a long time to come up with a solution to copy arguments from internal_f and then add a_bool argument:

import functools
from typing import TypeVar, Callable, Any
from typing_extensions import Concatenate, ParamSpec

I = ParamSpec("I")  # internal function
P = ParamSpec("P")  # public function
R = TypeVar("R")  # The return type of the internal function

def copy_inputs(internal_f: Callable[I, R]) -> Callable[[Callable[I, R]], Callable[Concatenate[bool, I], R]]:
    def wrap(public_f: Callable[P, R]) -> Callable[P, R]:
        @functools.wraps(public_f)
        def run(*args: P.args, **kwargs: P.kwargs) -> R:
            print(f"a_bool is {args[0]}")
            args = args[1:]
            internal_f(*args, **kwargs)
        return run
    return wrap

def internal_f(a_str: str, a_float: float) -> str:
    print(f"a_str is {a_str}, a_float is {a_float}")

@copy_inputs(internal_f)
def public_f(a_bool: bool, *args, **kwargs) -> str:
    ...

public_f(True, "a_string", 42.8)

I am not entirely sure I have everything type annotated correctly, but it does result in this on VSCode:

Also, I couldn't figure out how to get the argument named (i.e. rather than bool, it should show as a_bool: bool.

@ahuang11

@ahuang11 It looks like keyword parameter support by typing.Concatenate was postponed: https://peps.python.org/pep-0612/#concatenating-keyword-parameters... you can still accomplish this through a callback Protocol though: https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols

Does anyone know if this work is underway or not? I am definitely interested in making use of such a feature.

copying signature for external callers already works.

F = TypeVar('F', bound=Callable[..., Any])

class copy_signature(Generic[F]):
    def __init__(self, target: F) -> None: ...
    def __call__(self, wrapped: Callable[..., Any]) -> F: ...

def f(x: bool, *extra: int) -> str: ...

@copy_signature(f)
def test(*args, **kwargs):
    return f(*args, **kwargs)

reveal_type(test)  # Revealed type is 'def (x: bool, *extra: int) -> str'

The revealed type works as expected, but Mypy still reports the outer **kwargs as missing type annotation.

@lgetin that's pretty cool, but I think there's 2 problems with your example.

It's easy to forget when commenting on typing-related threads but most users probably aren't familiar enough with advanced typing constructs like TypeVars, Generics, etc. or even just with how to make their own decorators to come up with a solution like that. So unless they get lucky while searching StackOverflow this very common use-case will go unmet, resulting in badly-typed or untyped code even when the user would have wanted to type it.

The second problem is that even if everyone were to copy your copy_signature recipe, it's a bit 'too cute'. What I mean by that it uses metaprogramming that some of the most common type inference engines out in the wild aren't able to understand. For example, if I copy your code as-is into PyCharm and start invoking test the argument function signature assistance popup pane comes up as *args, **kwargs rather than x: bool, *extra: int the way it does for f.

I think a pretty good compromise solution would be to include either this exact recipe or an equivalent one in the typing module. This turns it into an official feature that nearly all code insight engines will then make sure to support, even if they have to special-case it.

EDIT: I also just realized that using a decorator like that there's no way to indicate that a method should keep the same signature as its closest parent in the MRO without explicitly referencing the parent. I think this is possibly the most common use-case for signature copying.

@matthewgdv

@lgetin that's pretty cool, but I think there's 2 problems with your example.

@matthewgdv that example isn't even functional, so there are definitely at least 3 problems with it 😉

I would like to propose the following implementations for a copy_signature-type decorator:

import functools
from typing import Any, Callable, Concatenate, ParamSpec, TypeVar

P = ParamSpec("P")
T = TypeVar("T")


def copy_function_signature(source: Callable[P, T]) -> Callable[[Callable], Callable[P, T]]:
    def wrapper(target: Callable) -> Callable[P, T]:
        @functools.wraps(source)
        def wrapped(*args: P.args, **kwargs: P.kwargs) -> T:
            return target(*args, **kwargs)

        return wrapped

    return wrapper


def copy_method_signature(source: Callable[Concatenate[Any, P], T]) -> Callable[[Callable], Callable[Concatenate[Any, P], T]]:
    def wrapper(target: Callable) -> Callable[Concatenate[Any, P], T]:
        @functools.wraps(source)
        def wrapped(self, *args: P.args, **kwargs: P.kwargs) -> T:
            return target(self, *args, **kwargs)

        return wrapped

    return wrapper


def f(x: bool, *extra: int) -> str:
    return str(...)


@copy_function_signature(f)
def test(*args, **kwargs):
    return f(*args, **kwargs)


class A:
    def foo(self, x: int, y: int, z: int) -> float:
        return float()


class B:
    @copy_method_signature(A.foo)
    def bar(self, *args, **kwargs):
        print(*args)


print(test(True, 1, 2, 3))  # Elipsis
B().bar(1, 2, 3)  # 1, 2, 3

reveal_type(test)  # Type of "test" is "(x: bool, *extra: int) -> str"
reveal_type(B.bar)  # Type of "B.bar" is "(Any, x: int, y: int, z: int) -> float"

Obviously we could make use of typing-extensions for Python versions <3.10. Curious to hear everyone's thoughts on this implementation and if there is truly a need for such functionality in typing.

@matthewgdv

@matthewgdv It wasn’t my example in the first place, it’s quoted from ilevkivskyi earlier in the thread (actually from over three years ago 😄). I just replied about it since it was recommended multiple times in the thread and also on Stack Overflow, but it didn’t seem to work as expected. I’m not very well-versed in the more advanced typing aspects of Python myself.

@ringohoffman

I would like to propose the following implementations for a copy_signature-type decorator:

@ringohoffman This caused a bunch of warnings when running Mypy in strict mode (via the --strict flag). I have attempted to make an updated version that fixes those warnings:

import functools
from collections.abc import Callable
from typing import Any, Concatenate, ParamSpec, TypeVar, reveal_type

P = ParamSpec("P")
T = TypeVar("T")


def copy_callable_signature(
    source: Callable[P, T]
) -> Callable[[Callable[..., T]], Callable[P, T]]:
    def wrapper(target: Callable[..., T]) -> Callable[P, T]:
        @functools.wraps(source)
        def wrapped(*args: P.args, **kwargs: P.kwargs) -> T:
            return target(*args, **kwargs)

        return wrapped

    return wrapper


def copy_method_signature(
    source: Callable[Concatenate[Any, P], T]
) -> Callable[[Callable[..., T]], Callable[Concatenate[Any, P], T]]:
    def wrapper(target: Callable[..., T]) -> Callable[Concatenate[Any, P], T]:
        @functools.wraps(source)
        def wrapped(self: Any, /, *args: P.args, **kwargs: P.kwargs) -> T:
            return target(self, *args, **kwargs)

        return wrapped

    return wrapper


def f(x: bool, *extra: int) -> str:
    return str(...)


@copy_callable_signature(f)
def test(*args, **kwargs):  # type: ignore[no-untyped-def] # copied signature
    return f(*args, **kwargs)


class A:
    def foo(self, x: int, y: int, z: int) -> float:
        return float()


class B:
    @copy_method_signature(A.foo)
    def bar(self, *args, **kwargs):  # type: ignore[no-untyped-def] # copied signature
        print(*args)


class Person:
    def __init__(self, given_name: str, surname: str):
        self.full_name = given_name + " " + surname

    def __repr__(self) -> str:
        return f"<{self.__class__.__name__}: {self.full_name}>"


@copy_callable_signature(Person)
def wrapper(*args, **kwargs):  # type: ignore[no-untyped-def] # copied signature
    return Person(*args, **kwargs)


print(test(True, 1, 2, 3))  # Ellipsis
B().bar(1, 2, 3)  # 1, 2, 3
print(wrapper("John", "Doe"))  # <Person: John Doe>

reveal_type(test)  # Type of "test" is "(x: bool, *extra: int) -> str"
reveal_type(B.bar)  # Type of "B.bar" is "(Any, x: int, y: int, z: int) -> float"
reveal_type(wrapper)  # Type of "wrapper" is "(given_name: str, surname: str) -> Person"

It runs without errors, and mypy --strict passes successfully. I also added another example copying the call signature of a class.

Interestingly, no-untyped-def errors will be raised for the decorated functions, so I have ignored them in the code. I wonder if this would actually count as a bug in Mypy. Since the revealed type shows that the function arguments and return type are fully typed, there’s no reason to raise an error about the function being untyped, right? 🤔

I've been experimenting with the copy_callable_signature from @Igetin's reply above in combination with Pydantic and the new typing.Self from Python 3.11. In the example below I am trying to create a classmethod that allows us to create a model with synthetic data without instantiating the class. However, the signature copying only works when all arguments are provided to .synthesize. Is there an obvious way to work around this limitation?

import functools
from collections.abc import Callable
from typing import ParamSpec, Self, TypeVar, reveal_type
from pydantic import BaseModel

P = ParamSpec("P")
T = TypeVar("T")


def copy_callable_signature(
    source: Callable[P, T]
) -> Callable[[Callable[..., T]], Callable[P, T]]:
    def wrapper(target: Callable[..., T]) -> Callable[P, T]:
        @functools.wraps(source)
        def wrapped(*args: P.args, **kwargs: P.kwargs) -> T:
            return target(*args, **kwargs)

        return wrapped

    return wrapper


class Message(BaseModel):
    author: str
    content: str

    @classmethod
    @copy_callable_signature(Self)
    def synthesize(cls, **kwargs):
        kwargs["author"] = kwargs.get("author", "Default author")
        kwargs["content"] = kwargs.get("content", "Default content")
        return cls(**kwargs)


message_1 = Message.synthesize(author="Moortiii") # Note that 'content' is missing
reveal_type(message_1) # Type of "message_1" is "Any"
reveal_type(message_1.synthesize) # Type of "message_1.synthesize" is "Any"

message_2 = Message.synthesize(author="Moortiii", content="Example message")
reveal_type(message_2) # Type of "message_2" is "Message"
reveal_type(message_2.synthesize) # Type of "message_2.synthesize" is "(*, author: str, content: str) -> Message"

print(message_1) # author='Moortiii' content='Default content'
print(message_2) # author='Moortiii' content='Example message'

@Moortiii

@Moortiii, it looks like you're not just trying to copy the signature. Instead what you need is to be able to modify the signature since some parameters in your new synthesize callable are optional even though they'd be required in the original. Unfortunately, with the typing tools we have today, this is not possible.

Also, on an unrelated note, kwargs["author"] = kwargs.get("author", "Default author") can be shortened to kwargs.setdefault("author", "Default author").

Ah yes, you're right. In the example above the type hint for synthesize would need to be (*, author: Optional[str] = None, content: Optional[str] = None) -> Message in order accurately represent the function. It's a shame to hear that this isn't currently possible. Are you aware of any ongoing efforts that may introduce this in the future? I've been unable to find any PEPs that are directly related to modifying the type of function arguments, not just the number of arguments itself.

Thinking out loud, I guess that even if we had a way to modify the arguments such that they all became Optional, this on its own would be confusing without a default value being specified in the type hint (which in our case we modify on kwargs directly, making it even more confusing). I imagine it would be difficult to provide an interface where users could specify an individual default value for each argument for usecases with different requirements than my own. Regardless, I feel like we're quickly stepping into the territory of the Sentinel PEP with regards to the default value, since None isn't always viable as a sentinel value either.

Indeed, kwargs.setdefault is a much nicer approach than what I had. Thanks for the heads up!

Hypothetically, depending on how keyword argument concatenation worked, that could be an avenue towards allowing for this. But I am not aware of any work being done in this regard.

The typing module is considering the inclusion of a copy_func_params decorator:
python/cpython#121693

(same as the copy_callable_signature proposed in this issue)

Navigation Menu

Repository navigation

Proposal: signature copying for kwargs. #270

43 remaining items

The Problem

P.S.

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Proposal: signature copying for kwargs. #270

Description

Activity

gvanrossum commented on Aug 26, 2016

JukkaL commented on Aug 26, 2016

dmoisset commented on Aug 31, 2016

JukkaL commented on Aug 31, 2016

dfee commented on Jun 13, 2018

alexmojaki commented on Jun 13, 2018

dfee commented on Jun 13, 2018

alexmojaki commented on Jun 13, 2018

dfee commented on Jun 13, 2018

dfee commented on Jun 14, 2018

JukkaL commented on Sep 6, 2018

ilevkivskyi commented on Sep 6, 2018

dfee commented on Sep 7, 2018

alexmojaki commented on Sep 17, 2018

43 remaining items

matthewgdv commented on Feb 16, 2021

rmorshea commented on Feb 17, 2021

USSX-Hares commented on Jun 22, 2022

The Problem

P.S.

ahuang11 commented on Oct 28, 2022

ringohoffman commented on Nov 30, 2022

Ige-fi commented on Dec 9, 2022

matthewgdv commented on Dec 9, 2022

ringohoffman commented on Dec 9, 2022

Ige-fi commented on Dec 9, 2022

Ige-fi commented on Dec 12, 2022

Moortiii commented on May 10, 2023

rmorshea commented on May 10, 2023

Moortiii commented on May 10, 2023

rmorshea commented on May 10, 2023

jaens commented on May 3, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions