fix(#3593): a failed drain schedule must not latch the pump, and the verdict must measure its mechanism

[rbuergi]

Refs #3593. 🚨 This does not claim the production wedge is fixed — see What this is not at the
end. It closes a third mechanism nobody had listed, and makes the remaining two tell themselves
apart.

The verdict asserted the one thing that mattered

#3615 made the stall report honest about three fields that were not measurements. One assertion
survived it. For the pump branch the line reads:

THE PUMP IS NOT TURNING: the drain flag is latched (a drain is scheduled on this hub's
TaskScheduler), drainsInFlight=0 … drainsInFlight=0 means the scheduled drain never ran: look
at the TaskScheduler that owes this hub a thread.

Nothing measured the parenthesis. drainsInFlight == 0 says no drain body is running; it does
not say one is queued. Two states read identically and have opposite owners:

	M1 — the scheduler holds it	M3 — nothing is outstanding
drainsInFlight	0	0
drainsScheduled − drainsStarted	> 0	0
Owner	the TaskScheduler	MessageService — this file

M3 was reachable, and it froze a hub for its whole life

draining = true is set inside the turn gate in KickDrain; the schedule happens outside it,
and Terminal() re-schedules with the latch still held. A throw from Task.Factory.StartNew at
either site left the flag set with nothing outstanding — and since KickDrain returns immediately
whenever the flag is set, the pump was then frozen for the life of the hub, silently, producing
exactly this issue's fingerprint (queue non-empty, draining=true, drainsInFlight=0, nothing ever
dequeued) with the verdict blaming a scheduler that had never been asked.

A real scheduler throws there: a completed ConcurrentExclusiveSchedulerPair, or an Orleans
activation torn down underneath the hub.

ScheduleDrainOne now releases the latch and logs an Error naming the scheduler and the queue
depth, so the invariant "draining ⇒ a drain is running or queued" holds by construction and
every verdict that reads it is entitled to.

🚨 It does not fall back to another scheduler. The turn scheduler is the hub's serialisation
guarantee — on a root grain hub it is the activation scheduler — so running a turn elsewhere would
break the actor model to keep a queue moving. Releasing the latch is the honest recovery: the next
post tries again and reports again, instead of the pump going dark.

And the verdict now measures its mechanism

drainsScheduled / drainsStarted are counted at the two ends of the hand-off; the snapshot
carries the difference; the line picks its MECHANISM: clause from the counters instead of
asserting one:

• drainsInFlight > 0 → M2: a drain body is running and blocked before the dequeue.
• drainsAwaitingScheduler > 0 → M1: the scheduler accepted work and is not running it. On
  TaskScheduler.Default that is pool starvation or a local-LIFO stall; on a root grain hub it
  is a wedged or deactivated ActivationTaskScheduler.
• both zero → M3: a defect in ScheduleDrainOne, which the change above makes unreachable.

Tests

DisposalStallNamesThePumpTest — the suite that already builds this exact state:

• the existing pump-verdict test gains drainsAwaitingScheduler=1, the the turn scheduler ACCEPTED
  mechanism clause, and NotContain("defect in MessageService.ScheduleDrainOne") — the pump did
  its job there, and blaming it for a scheduler that will not run work is the same misattribution as
  blaming a child;
• ASchedulerThatRefusesADrain_DoesNotLeaveThePumpLatched (new) — a scheduler that throws on
  queue, then accepts again; the hub must answer the next request. It asserts the refusal actually
  happened first, so it cannot pass over a scheduler that quietly worked.

Falsified: with the latch release removed, the new test fails — the post after recovery is never
dequeued and the request times out. Both pass with it. No SemaphoreSlim, no Task.Delay, no
bridge: both fake schedulers are plain TaskScheduler subclasses over a Lock.

Verification

dotnet build -c Release -warnaserror, one project per invocation: MeshWeaver.Messaging.Hub,
MeshWeaver.Documentation, MeshWeaver.Messaging.Hub.Test, MeshWeaver.Documentation.Test —
0 Warning(s), 0 Error(s) each. Tests below.

Docs

• Doc/Architecture/DisposalStallVerdicts — M3 added to the mechanism table, the sample verdict
  updated, and the Orleans clause corrected (see below).
• What's New: A latched pump that scheduled nothing (Category: Fix).

🚨 What this is not

It is almost certainly not what the 47 production reports were, and the doc says so.
Task.Factory.StartNew on TaskScheduler.Default does not throw, and a hosted hub is built from a
fresh MessageHubConfiguration (IServiceProvider.CreateMessageHub) — nothing copies a
parent's TaskScheduler, which is exactly what WithTaskScheduler's own contract asks for ("use
the grain's scheduler ONLY for the root grain hub"). So the sync/* hubs in that incident run on
the default pool scheduler, where M3 is unreachable and a dead activation scheduler cannot reach
them either.

That narrows the candidate set for the incident to M1-by-pool-starvation or M2 and leaves #3593
open — the same call #3615 made. What changes is that the next occurrence prints which one it is,
and that a real latch leak with the identical fingerprint can no longer be one of them.

[Copilot]

🟢 Approval recommended

The behavioral changes are covered by targeted tests and documentation, with only a minor in-code comment correction suggested.

Pull request overview

This PR addresses a hub message-pump edge case where a failed drain scheduling attempt could leave the pump latched (draining=true) with no drain running or queued, permanently preventing future dequeues; it also strengthens the disposal-stall verdict so it reports the measured mechanism (scheduler accepted-but-didn’t-run vs. pump latched-with-nothing-outstanding).

Changes:

• Make MessageService.ScheduleDrainOne release the drain latch (and log) when Task.Factory.StartNew(..., turnScheduler) throws, preventing a permanent pump freeze.
• Add drain hand-off counters (drainsScheduled, drainsStarted) and plumb drainsAwaitingScheduler through snapshots so MessageHub can select a “MECHANISM:” clause based on measurements.
• Extend/introduce tests and documentation to cover the new mechanism discrimination and the latch-release behavior.

File summaries

File	Description
test/MeshWeaver.Messaging.Hub.Test/DisposalStallNamesThePumpTest.cs	Extends the existing pump-verdict assertions and adds a new regression test ensuring a refused schedule does not leave the pump latched.
src/MeshWeaver.Messaging.Hub/MessageService.cs	Adds drain hand-off counters and makes ScheduleDrainOne release the latch + log when scheduling throws.
src/MeshWeaver.Messaging.Hub/MessageHub.cs	Updates disposal stall verdict wording and mechanism selection to use drainsAwaitingScheduler as a measurement.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-09-a-latched-pump-that-scheduled-nothing.md	Adds a What’s New entry describing the latch-leak fix and improved mechanism reporting.
src/MeshWeaver.Documentation/Data/Architecture/DisposalStallVerdicts.md	Updates the architecture doc to include the new M3 mechanism and the new measured discriminator fields.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Test Results (shard 1)

496 tests   496 ✅  47s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 07c5ace.

[github-actions]

Test Results (shard 0)

254 tests   254 ✅  2m 15s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 07c5ace.

[github-actions]

Test Results (shard 3)

389 tests   389 ✅  50s ⏱️
  2 suites    0 💤
  2 files      0 ❌

Results for commit 07c5ace.

[github-actions]

Test Results (shard 4)

1 861 tests   1 861 ✅  2m 37s ⏱️
    4 suites      0 💤
    4 files        0 ❌

Results for commit 07c5ace.

[github-actions]

Test Results (shard 2)

1 142 tests   1 140 ✅  3m 36s ⏱️
    3 suites      2 💤
    3 files        0 ❌

Results for commit 07c5ace.

[github-actions]

Test Results (shard 5)

    6 files      6 suites   4m 38s ⏱️
2 499 tests 2 307 ✅ 192 💤 0 ❌
2 500 runs  2 308 ✅ 192 💤 0 ❌

Results for commit 07c5ace.

[github-actions]

Test Results

   17 files     17 suites   14m 44s ⏱️
6 641 tests 6 447 ✅ 194 💤 0 ❌
6 642 runs  6 448 ✅ 194 💤 0 ❌

Results for commit 07c5ace.