Skip to content

IoPool: a leaf between Subscribe() and its ThreadPool prologue is invisible to Drain()'s grace #4555

Closed

Description

@rbuergi

The window

Between a caller's Subscribe() returning and the ThreadPool running the leaf's prologue, the leaf
is counted by nothing — not _gateUsers, not _inFlight, not CurrentlyWaiting. The caller
believes the work is queued; the pool does not know it exists.

Three of the four entry points have such a window, and the consequence is not the same in each:

entry point region taken consequence of the gap
Invoke / InvokeStream inside the SubscribeOn'd body the leaf is cancelled at its gate wait when it finally arrives, too late even to be named — accepted work discarded silently
SubscribeThroughPool (setup leaf) outer region released in the subscribe's finally, before the leaf runs milder: the drain registration is already armed (#4524), and the leaf re-checks its linked token before source.Subscribe(observer), so the leg is refused and terminated rather than run after teardown. The use-after-unload precondition is not reopened; what is lost is that the drain does not wait for that subscribe
InvokeBlocking subscriber's thread, spanning the whole leaf (closed by the ContinueWith) none
// Invoke / InvokeStream — the admission is inside the scheduled body
=> Observable.FromAsync(async subscriberCt =>
   {
       if (!TryEnterGateRegion())          // ← on a POOL thread
           throw new OperationCanceledException(DisposedMessage);
       ...
   }).SubscribeOn(TaskPoolScheduler.Default);

// SubscribeThroughPoolCore — outer region taken synchronously, but released before the setup leaf runs
try { ... var setup = Observable.FromAsync(...).SubscribeOn(TaskPoolScheduler.Default).Subscribe(...); ... }
finally { LeaveGateRegion(); }             // ← released at the end of the synchronous subscribe

Why it matters to Drain()

Drain()'s grace waits while anything is outstanding and restarts on every completion. If the
outstanding count reaches zero inside this window, the grace ends and the pool token is
cancelled. For Invoke/InvokeStream the invisible leaf is then cancelled at its gate wait — too
late to appear in LeavesCancelledAfterGrace / CancelledLeafSites, which are captured before the
cancel. That is accepted work discarded without being named, the same contract violation #3291
addressed from the other direction (teardown lets accepted work finish and names what it had to
stop
).

What is measured and what is not

#4541 was a different defect in the same method — the grace read progress off the live admission
census, so an arrival cancelled out a completion and the per-completion grace collapsed into one
total budget. That one is reproduced deterministically and fixed in #4556. Fixing it does not
close this window; it only removes the masking. Both mechanisms produce the same observable symptom
(a queued leaf cancelled during a drain), which is why they are worth keeping apart.

The likely shape of a fix

Take the admission on the subscriber's thread, the way InvokeBlocking already does, and hold it
until the leaf terminates — a small Observable.Create wrapper that enters the region, subscribes
the SubscribeOn'd leaf, and leaves the region exactly once on terminal-or-dispose. Then "accepted"
means the same thing on every entry point, and a leaf whose Subscribe() returned before the drain
finished its grace is either run to completion or named as cancelled.

Two things to weigh, neither blocking:

  1. It moves the refusal terminal (OperationCanceledException for a disposed/draining pool) onto
    the subscriber's thread. That matches what the build-time refusal already does — Cancelled<T>()
    is Observable.Throw on the immediate scheduler — but it is adjacent to IoPool: a late drain registration no longer runs a pooled teardown on the subscriber's thread (#4524) #4533 / A pooled leg REFUSED after Drain()/Dispose() terminates inside Subscribe() on the subscriber's thread — OrderedRouteDispatcher.DrainNext then recurses through its queue #4530, which are
    about which thread a terminal runs on, so it should land with those in view rather than against
    them.
  2. It touches the hottest entry points in the system, so it wants its own PR and its own review
    rather than riding along with a drain fix.

Deliberately filed rather than fixed in #4556: that PR is scoped to Drain() and its counters, and
the subscribe path is being edited concurrently (#4530 / #4545).

Refs #4541, #3291, #4524, #4533, #4530, #4545, #4556.

Activity

  1. changed the title [-]IoPool: a leaf between Subscribe() and its ThreadPool prologue is invisible to Drain()'s grace (Invoke / InvokeStream)[/-] [+]IoPool: a leaf between Subscribe() and its ThreadPool prologue is invisible to Drain()'s grace[/+] on Sep 17, 2026
  2. rbuergi commented on Sep 17, 2026

    @rbuergi
    ContributorAuthor

    Scope corrected. The original body claimed this window was limited to Invoke/InvokeStream and that SubscribeThroughPool did not have it — that was wrong, and the Copilot review on #4556 caught it.

    SubscribeThroughPoolCore takes its outer gate region synchronously on the subscriber's thread, but releases it in the subscribe's finally, while the setup leaf enters a second region later on a pool thread. So a pooled subscribe that returned before the drain can also be invisible until its prologue runs.

    The consequence is milder there and worth stating precisely rather than lumping together: the drain registration is already armed (#4524) and the setup leaf re-checks its linked token before source.Subscribe(observer), so a leg caught in that gap is refused and terminated, not run after teardown — the use-after-unload precondition is not reopened. What is lost is that the drain does not wait for that subscribe.

    InvokeBlocking remains the one path with no gap: its region is taken on the subscriber's thread and spans the whole leaf.

  3. added a commit that references this issue on Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions