Change create_task type

[sobolevn]

I've manually tested that Awaitables are not supported:

import asyncio

class My:
    def __await__(self):
        async def inner():
            print('here')
        return inner

async def main():
  await asyncio.create_task(My())

asyncio.run(main())

Output:

Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/build/ex.py", line 15, in <module>
    asyncio.run(main())
  File "/Users/sobolev/.pyenv/versions/3.9.9/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/sobolev/.pyenv/versions/3.9.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/Users/sobolev/Desktop/cpython/build/ex.py", line 13, in main
    await asyncio.create_task(My())
  File "/Users/sobolev/.pyenv/versions/3.9.9/lib/python3.9/asyncio/tasks.py", line 361, in create_task
    task = loop.create_task(coro)
  File "/Users/sobolev/.pyenv/versions/3.9.9/lib/python3.9/asyncio/base_events.py", line 433, in create_task
    task = tasks.Task(coro, loop=self, name=name)
TypeError: a coroutine was expected, got <__main__.My object at 0x10c38fe20>

But, Generator objects are also supported:

import asyncio
def gen():
   i = yield
   print(1, i)  # prints: `1 None`

async def main():
  await asyncio.create_task(gen())

asyncio.run(main())

Closes #6770

[sobolevn]

Notice that _TaskYieldType was not used.

[Akuli]

This doesn't seem right. Event loops typically use yields to communicate between tasks and the loop, so the generator must yield objects of a specific type.

[sobolevn]

Some experiments:

import asyncio
def gen():
   i = yield
   print(1, i)  # prints: `1, None`
   return 'a'

async def main():
  print('res', await asyncio.create_task(gen()))  # prints: `res, a`

asyncio.run(main())

And:

import asyncio
def gen():
   i = yield
   print(1, i)  # prints: `1, None`
   yield 'a'

async def main():
  print('res', await asyncio.create_task(gen()))

asyncio.run(main())

1 None
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/build/ex.py", line 10, in <module>
    asyncio.run(main())
  File "/Users/sobolev/.pyenv/versions/3.9.9/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/sobolev/.pyenv/versions/3.9.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/Users/sobolev/Desktop/cpython/build/ex.py", line 8, in main
    print('res', await asyncio.create_task(gen()))  # prints: `res, a`
  File "/Users/sobolev/Desktop/cpython/build/ex.py", line 5, in gen
    yield 'a'
RuntimeError: Task got bad yield: 'a'

[sobolevn]

Ok, we have some problems:

self._navigationPromise = self._loop.create_task(asyncio.wait([
    self._lifecycleCompletePromise,
    self._createTimeoutPromise(),
], return_when=concurrent.futures.FIRST_COMPLETED))

Here asyncio.wait returns Future, but now self._loop.create_task() wants Coroutine. Future is not compatible with Coroutine. 🤔

[Akuli]

Doesn't look like a Future to me:

Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> asyncio.wait([])
<coroutine object wait at 0x7f90e8cef140>

[sobolevn]

@Akuli sorry, I had to clarify that I'm talking about typeshed's definition:

typeshed/stdlib/asyncio/tasks.pyi

Lines 235 to 239 in 505ea72

	def wait(fs: Iterable[_FT], *, timeout: float | None = ..., return_when: str = ...) -> Future[tuple[set[_FT], set[_FT]]]: ... # type: ignore[misc]
	@overload
	def wait(
	fs: Iterable[Awaitable[_T]], *, timeout: float | None = ..., return_when: str = ...
	) -> Future[tuple[set[Task[_T]], set[Task[_T]]]]: ...

[Akuli]

A few observations:

• The coro argument gets passed to Task.__init__, so that should be changed too.
• Task.__init__ uses asyncio.coroutines.iscoroutine, which is an optimized isinstance(obj, asyncio.coroutines._COROUTINE_TYPES), not to be confused with inspect.iscoroutine which is isinstance(obj, types.CoroutineType).
• asyncio is a mess :(

[sobolevn]

Ok, I have no idea what is going on:

• Why some definitions are async def, while others are sync def that return Future?
• Why do they return Future, why not Coroutine?
• Why stubtest allows this?

[sobolevn]

Ok, my plan is to sync all async def in source code and async def in typeshed (unless there are some problems that I don't know about).

[hauntsaninja]

Probably shouldn't merge this as is, given the false positives. The one in my code could be potentially worked around by changing AsyncIterator.__anext__ to return a Coroutine.

Re stubtest:
It's one of many tricks stubtest doesn't know about / mypy itself largely treats def f() -> Coroutine[Any, Any, int] the same as async def f() -> int so it's not something you're forced to handle. Can't remember if I tried adding support for this to stubtest; there were several things I chose not to add based on typeshed error counts. Might have been affected by older Pythons, e.g. on Python 3.5 several functions are @coroutine not async def.

[sobolevn]

Totally do not merge this as-is, I will spend several days fixing all the async def / def stuff. And hopefully after that we can return to this issue.

[srittau]

I've converted this to a draft for now. Please mark as "ready for review" when it is. :)

[sobolevn]

I've changed all things I could find to be async def:

typeshed/stdlib/asyncio/tasks.pyi

Lines 235 to 239 in 7bc9c16

	async def wait(fs: Iterable[_FT], *, timeout: float | None = ..., return_when: str = ...) -> tuple[set[_FT], set[_FT]]: ... # type: ignore[misc]
	@overload
	async def wait(
	fs: Iterable[Awaitable[_T]], *, timeout: float | None = ..., return_when: str = ...
	) -> tuple[set[Task[_T]], set[Task[_T]]]: ...

But, mypy-primer found some new ones 🤔

[sobolevn]

Analysis:

• homeasistant annotation is not correct, it uses Awaitable, which is not supported by create_task: https://github.com/home-assistant/core/blob/19b7454161046d68dc388e7ff1ffdd0573855449/homeassistant/core.py#L439 I will send a PR with the fix to their repo 👍
• @hauntsaninja, what do you think about https://github.com/hauntsaninja/boostedblob/blob/master/boostedblob/boost.py#L547 ?
• Argument 1 to "create_task" has incompatible type "Awaitable[Sequence[Any]]"; expected "Union[Generator[Any, None, <nothing>], Coroutine[Any, Any, <nothing>]]" seems correct to me

This error does not look good to me: https://github.com/Yelp/paasta/blob/5e504597ea5a2af952a12785808e523b53b7c01d/paasta_tools/instance/kubernetes.py#L545 🤔

Aha! Found the reason: the decorator annotation is not quite correct: https://github.com/Yelp/paasta/blob/5e504597ea5a2af952a12785808e523b53b7c01d/paasta_tools/async_utils.py#L93-L97

[srittau]

@sobolevn If I understand correctly, in your opinion the primer diffs all show genuine problems?

[sobolevn]

@srittau yes, I think so! I've opened #7105 to fix 2.

[srittau]

Ok, I'll have a look once the merge conflicts are resolved.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

core (https://github.com/home-assistant/core)
+ homeassistant/core.py:439: error: Argument 1 to "create_task" of "AbstractEventLoop" has incompatible type "Awaitable[_R]"; expected "Union[Coroutine[Any, Any, _R], Generator[Any, None, _R]]"  [arg-type]
+ homeassistant/core.py:470: error: Need type annotation for "task"  [var-annotated]
+ homeassistant/core.py:470: error: Argument 1 to "create_task" of "AbstractEventLoop" has incompatible type "Awaitable[_R]"; expected "Union[Coroutine[Any, Any, <nothing>], Generator[Any, None, <nothing>]]"  [arg-type]

paasta (https://github.com/yelp/paasta)
- paasta_tools/instance/kubernetes.py:553: error: Argument 1 to "append" of "list" has incompatible type "Task[Sequence[Any]]"; expected "Future[Dict[str, Any]]"
+ paasta_tools/instance/kubernetes.py:545: error: Need type annotation for "pods_task"
+ paasta_tools/instance/kubernetes.py:546: error: Argument 1 to "create_task" has incompatible type "Awaitable[Sequence[Any]]"; expected "Union[Generator[Any, None, <nothing>], Coroutine[Any, Any, <nothing>]]"
+ paasta_tools/instance/kubernetes.py:1017: error: Need type annotation for "pods_task"
+ paasta_tools/instance/kubernetes.py:1018: error: Argument 1 to "create_task" has incompatible type "Awaitable[Sequence[Any]]"; expected "Union[Generator[Any, None, <nothing>], Coroutine[Any, Any, <nothing>]]"
+ paasta_tools/instance/kubernetes.py:1202: error: Need type annotation for "pods_task"
+ paasta_tools/instance/kubernetes.py:1203: error: Argument 1 to "create_task" has incompatible type "Awaitable[Sequence[Any]]"; expected "Union[Generator[Any, None, <nothing>], Coroutine[Any, Any, <nothing>]]"

[srittau]

Nit for future reference: Coroutine should now be imported from collections.abc.

[sobolevn]

Thanks everyone 🎉