Repository navigation
IoPool: a leaf between Subscribe() and its ThreadPool prologue is invisible to Drain()'s grace #4555
Description
The window
Between a caller's Subscribe() returning and the ThreadPool running the leaf's prologue, the leaf
is counted by nothing — not _gateUsers, not _inFlight, not CurrentlyWaiting. The caller
believes the work is queued; the pool does not know it exists.
Three of the four entry points have such a window, and the consequence is not the same in each:
| entry point | region taken | consequence of the gap |
|---|---|---|
Invoke / InvokeStream |
inside the SubscribeOn'd body |
the leaf is cancelled at its gate wait when it finally arrives, too late even to be named — accepted work discarded silently |
SubscribeThroughPool (setup leaf) |
outer region released in the subscribe's finally, before the leaf runs |
milder: the drain registration is already armed (#4524), and the leaf re-checks its linked token before source.Subscribe(observer), so the leg is refused and terminated rather than run after teardown. The use-after-unload precondition is not reopened; what is lost is that the drain does not wait for that subscribe |
InvokeBlocking |
subscriber's thread, spanning the whole leaf (closed by the ContinueWith) |
none |
// Invoke / InvokeStream — the admission is inside the scheduled body
=> Observable.FromAsync(async subscriberCt =>
{
if (!TryEnterGateRegion()) // ← on a POOL thread
throw new OperationCanceledException(DisposedMessage);
...
}).SubscribeOn(TaskPoolScheduler.Default);
// SubscribeThroughPoolCore — outer region taken synchronously, but released before the setup leaf runs
try { ... var setup = Observable.FromAsync(...).SubscribeOn(TaskPoolScheduler.Default).Subscribe(...); ... }
finally { LeaveGateRegion(); } // ← released at the end of the synchronous subscribeWhy it matters to Drain()
Drain()'s grace waits while anything is outstanding and restarts on every completion. If the
outstanding count reaches zero inside this window, the grace ends and the pool token is
cancelled. For Invoke/InvokeStream the invisible leaf is then cancelled at its gate wait — too
late to appear in LeavesCancelledAfterGrace / CancelledLeafSites, which are captured before the
cancel. That is accepted work discarded without being named, the same contract violation #3291
addressed from the other direction (teardown lets accepted work finish and names what it had to
stop).
What is measured and what is not
- Structural / readable from the code: the window itself, and which entry points have it.
- NOT measured: that it has ever fired. I did not reproduce a leaf being silently cancelled
through it, and I am not claiming it caused the failure in Drain() can cancel pooled leaves that were accepted but still QUEUED for a slot — IoPoolTest grace-restart test failed 1 of 5 under CPU saturation #4541.
#4541 was a different defect in the same method — the grace read progress off the live admission
census, so an arrival cancelled out a completion and the per-completion grace collapsed into one
total budget. That one is reproduced deterministically and fixed in #4556. Fixing it does not
close this window; it only removes the masking. Both mechanisms produce the same observable symptom
(a queued leaf cancelled during a drain), which is why they are worth keeping apart.
The likely shape of a fix
Take the admission on the subscriber's thread, the way InvokeBlocking already does, and hold it
until the leaf terminates — a small Observable.Create wrapper that enters the region, subscribes
the SubscribeOn'd leaf, and leaves the region exactly once on terminal-or-dispose. Then "accepted"
means the same thing on every entry point, and a leaf whose Subscribe() returned before the drain
finished its grace is either run to completion or named as cancelled.
Two things to weigh, neither blocking:
- It moves the refusal terminal (
OperationCanceledExceptionfor a disposed/draining pool) onto
the subscriber's thread. That matches what the build-time refusal already does —Cancelled<T>()
isObservable.Throwon the immediate scheduler — but it is adjacent to IoPool: a late drain registration no longer runs a pooled teardown on the subscriber's thread (#4524) #4533 / A pooled leg REFUSED after Drain()/Dispose() terminates inside Subscribe() on the subscriber's thread — OrderedRouteDispatcher.DrainNext then recurses through its queue #4530, which are
about which thread a terminal runs on, so it should land with those in view rather than against
them. - It touches the hottest entry points in the system, so it wants its own PR and its own review
rather than riding along with a drain fix.
Deliberately filed rather than fixed in #4556: that PR is scoped to Drain() and its counters, and
the subscribe path is being edited concurrently (#4530 / #4545).
Activity
[-]IoPool: a leaf between Subscribe() and its ThreadPool prologue is invisible to Drain()'s grace (Invoke / InvokeStream)[/-][+]IoPool: a leaf between Subscribe() and its ThreadPool prologue is invisible to Drain()'s grace[/+]rbuergi commentedon Sep 17, 2026
Scope corrected. The original body claimed this window was limited to
Invoke/InvokeStreamand thatSubscribeThroughPooldid not have it — that was wrong, and the Copilot review on #4556 caught it.SubscribeThroughPoolCoretakes its outer gate region synchronously on the subscriber's thread, but releases it in the subscribe'sfinally, while the setup leaf enters a second region later on a pool thread. So a pooled subscribe that returned before the drain can also be invisible until its prologue runs.The consequence is milder there and worth stating precisely rather than lumping together: the drain registration is already armed (#4524) and the setup leaf re-checks its linked token before
source.Subscribe(observer), so a leg caught in that gap is refused and terminated, not run after teardown — the use-after-unload precondition is not reopened. What is lost is that the drain does not wait for that subscribe.InvokeBlockingremains the one path with no gap: its region is taken on the subscriber's thread and spans the whole leaf.Address the review on #4556: an unfailable precondition, and a wrong …