Drain() can cancel pooled leaves that were accepted but still QUEUED for a slot — IoPoolTest grace-restart test failed 1 of 5 under CPU saturation #4541

What failed

IoPoolTest.Drain_restartsTheGraceOnEveryCompletion_SoABurstOfShortLeavesIsNeverCancelled (test/MeshWeaver.Hosting.Test/IoPoolTest.cs) failed once:

Expected value to be 3 … but found 1

That is the assertion completedLeaves.Should().Be(3, "every leaf finished on its own") — so two of three leaves that were subscribed before Drain() began did not complete.

Observed in a local run of #4533's branch on 2026-09-16. It passed in the four runs before it. It has not been re-run, deliberately.

🚨 This is unattributed, not a flake

Neither #4533's diff nor the probe edits made in that worktree can reach it — the test uses only Invoke and Drain. A test that fails on a diff that cannot reach it has a cause of its own.

The test, and how little room it has

var grace = TimeSpan.FromMilliseconds(500);
using var pool = new IoPool(1, IoPool.DefaultDrainTimeout, grace);   // ONE slot
// three leaves, each: await Task.Delay(300, ct)
await firstAdmitted.Should().Within(Timeout5).Emit();
var residual = pool.Drain();

With one slot the three 300 ms leaves run serially, so leaves 2 and 3 are queued waiting for the slot when Drain() starts. The grace restarts on each completion — so each queued leaf has about 200 ms (500 − 300) to be picked up and started, or the grace expires.

Condition at the time (measured)

The machine was CPU-saturated: load average ~29 on 18 cores, top reporting 79 % user + 10 % sys, 11 % idle, from seven concurrent sessions' Release builds and a local in-mesh tester container. Under that load, a ThreadPool work item can plausibly wait longer than 200 ms to start.

That is the condition, not the explanation. Load does not create a defect; it widens a window that is already there.

Probable cause — inferred from the code, NOT measured

Drain()'s grace is progress-based on the pool's gate users. A leaf that has been subscribed but is still queued for the slot — not yet started on the ThreadPool — appears to be invisible to that progress signal. If so, the drain treats "nothing has completed recently" as "stuck" and cancels work that was accepted before the drain began and was only waiting its turn.

If that reading is right, it conflicts with the principle #3291 established — teardown lets accepted work finish and names what it had to stop. A leaf queued for a slot is accepted work.

What is NOT the fix

• Raising the grace, or lengthening the leaves' margin. That buys slack against load and leaves the question unanswered. A bound that holds only on an idle machine is a band-aid (AGENTS.md).
• Re-running until it passes. It passed four times already; that is what hid it.

Suggested next step

Establish the mechanism by widening the window deterministically, not by sweeping: park a queued leaf's admission for longer than one grace (a seam, or a saturated ThreadPool), with no competing CPU, and assert whether Drain() cancels it. If it does, the decision is whether a queued-but-accepted leaf should count as progress for the grace, or be refused at the front door before it is accepted — either is coherent; cancelling it silently after acceptance is not.

Found while verifying #4533 (#4524). Related: #4530 (a refused leg terminating on the subscriber's thread — a different path), #3291, #4448.