Skip to content

Commit 4d05554

Browse files
committed
Narrow forkserver hang to async_main outer tn
Fourth diagnostic pass — instrument `_worker`'s fork-child branch (`pre child_target()` / `child_ target RETURNED rc=N` / `about to os._exit(rc)`) and `_trio_main` boundaries (`about to trio.run` / `trio.run RETURNED NORMALLY` / `FINALLY`). Test config: depth=1/breadth=2 = 1 root + 14 forked = 15 actors total. Fresh-run results, - **9 processes complete the full flow**: `trio.run RETURNED NORMALLY` → `child_target RETURNED rc=0` → `os._exit(0)`. These are tree LEAVES (errorers) plus their direct parents (depth-0 spawners) — they actually exit - **5 processes stuck INSIDE `trio.run(trio_ main)`**: hit "about to trio.run" but never see "trio.run RETURNED NORMALLY". These are root + top-level spawners + one intermediate The deadlock is in `async_main` itself, NOT the peer-channel loops. Specifically, the outer `async with root_tn:` in `async_main` never exits for the 5 stuck actors, so the cascade wedges: trio.run never returns → _trio_main finally never runs → _worker never reaches os._exit(rc) → process never dies → parent's _ForkedProc.wait() blocks → parent's nursery hangs → parent's async_main hangs → (recurse up) The precise new question: **what task in the 5 stuck actors' `async_main` never completes?** Candidates: 1. shielded parent-chan `process_messages` task in `root_tn` — but we cancel it via `_parent_chan_cs.cancel()` in `Actor.cancel()`, which only runs during `open_root_actor.__aexit__`, which itself runs only after `async_main`'s outer unwind — which doesn't happen. So the shield isn't broken in this path. 2. `actor_nursery._join_procs.wait()` or similar inline in the backend `*_proc` flow. 3. `_ForkedProc.wait()` on a grandchild that DID exit — but pidfd_open watch didn't fire (race between `pidfd_open` and the child exiting?). Most specific next probe: add DIAG around `_ForkedProc.wait()` enter/exit to see whether pidfd-based wait returns for every grandchild exit. If a stuck parent's `_ForkedProc.wait()` never returns despite its child exiting → pidfd mechanism has a race bug under nested forkserver. Asymmetry observed in the cascade tree: some d=0 spawners exit cleanly, others stick, even though they started identically. Not purely depth- determined — some race condition in nursery teardown when multiple siblings error simultaneously. No code changes — diagnosis-only. (this commit msg was generated in some part by [`claude-code`][claude-code-gh]) [claude-code-gh]: https://github.com/anthropics/claude-code
1 parent ab86f76 commit 4d05554

1 file changed

Lines changed: 95 additions & 0 deletions

File tree

ai/conc-anal/subint_forkserver_test_cancellation_leak_issue.md

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -540,6 +540,101 @@ their owning actor's `Actor.cancel()` never runs.
540540
The recvs are fine — they're just parked because
541541
nothing is telling them to stop.
542542

543+
## Update — 2026-04-23 (very late): leaves exit, middle actors stuck in `trio.run`
544+
545+
Yet another instrumentation pass — this time
546+
printing at:
547+
548+
- `_worker` child branch: `pre child_target()` /
549+
`child_target RETURNED rc=N` / `about to
550+
os._exit(rc)`
551+
- `_trio_main`: `about to trio.run` /
552+
`trio.run RETURNED NORMALLY` / `FINALLY`
553+
554+
**Fresh-run results** (`test_nested_multierrors[
555+
subint_forkserver]`, depth=1/breadth=2, 1 root + 14
556+
forked = 15 actors total):
557+
558+
- **9 processes completed the full flow**
559+
`trio.run RETURNED NORMALLY` → `child_target
560+
RETURNED rc=0``about to os._exit(0)`. These
561+
are the LEAVES of the tree (errorer actors) plus
562+
their direct parents (depth-0 spawners). They
563+
actually exit their processes.
564+
- **5 processes are stuck INSIDE `trio.run(trio_main)`**
565+
— they hit "about to trio.run" but NEVER see
566+
"trio.run RETURNED NORMALLY". These are root +
567+
top-level spawners + one intermediate.
568+
569+
**What this means:** `async_main` itself is the
570+
deadlock holder, not the peer-channel loops.
571+
Specifically, the outer `async with root_tn:` in
572+
`async_main` never exits for the 5 stuck actors.
573+
Their `trio.run` never returns → `_trio_main`
574+
catch/finally never runs → `_worker` never reaches
575+
`os._exit(rc)` → the PROCESS never dies → its
576+
parent's `_ForkedProc.wait()` blocks → parent's
577+
nursery hangs → parent's `async_main` hangs → ...
578+
579+
### The new precise question
580+
581+
**What task in the 5 stuck actors' `async_main`
582+
never completes?** Candidates:
583+
584+
1. The shielded parent-chan `process_messages`
585+
task in `root_tn` — but we explicitly cancel it
586+
via `_parent_chan_cs.cancel()` in `Actor.cancel()`.
587+
However, `Actor.cancel()` only runs during
588+
`open_root_actor.__aexit__`, which itself runs
589+
only after `async_main`'s outer unwind — which
590+
doesn't happen. So the shield isn't broken.
591+
592+
2. `await actor_nursery._join_procs.wait()` or
593+
similar in the inline backend `*_proc` flow.
594+
595+
3. `_ForkedProc.wait()` on a grandchild that
596+
actually DID exit — but the pidfd_open watch
597+
didn't fire for some reason (race between
598+
pidfd_open and the child exiting?).
599+
600+
The most specific next probe: **add DIAG around
601+
`_ForkedProc.wait()` enter/exit** to see whether
602+
the pidfd-based wait returns for every grandchild
603+
exit. If a stuck parent's `_ForkedProc.wait()`
604+
NEVER returns despite its child exiting, the
605+
pidfd mechanism has a race bug under nested
606+
forkserver.
607+
608+
Alternative probe: instrument `async_main`'s outer
609+
nursery exits to find which nursery's `__aexit__`
610+
is stuck, drilling down from `trio.run` to the
611+
specific `async with` that never completes.
612+
613+
### Cascade summary (updated tree view)
614+
615+
```
616+
ROOT (pytest) STUCK in trio.run
617+
├── top_0 (spawner, d=1) STUCK in trio.run
618+
│ ├── spawner_0_d1_0 (d=0) exited (os._exit 0)
619+
│ │ ├── errorer_0_0 exited (os._exit 0)
620+
│ │ └── errorer_0_1 exited (os._exit 0)
621+
│ └── spawner_0_d1_1 (d=0) exited (os._exit 0)
622+
│ ├── errorer_0_2 exited (os._exit 0)
623+
│ └── errorer_0_3 exited (os._exit 0)
624+
└── top_1 (spawner, d=1) STUCK in trio.run
625+
├── spawner_1_d1_0 (d=0) STUCK in trio.run (sibling race?)
626+
│ ├── errorer_1_0 exited
627+
│ └── errorer_1_1 exited
628+
└── spawner_1_d1_1 (d=0) STUCK in trio.run
629+
├── errorer_1_2 exited
630+
└── errorer_1_3 exited
631+
```
632+
633+
Grandchildren (d=0 spawners) exit OR stick —
634+
asymmetric. Not purely depth-determined. Some race
635+
condition in nursery teardown when multiple
636+
siblings error simultaneously.
637+
543638
## Stopgap (landed)
544639

545640
`test_nested_multierrors` skip-marked under

0 commit comments

Comments
 (0)