Repository navigation
Pod shutdown stalls: hosted sync hubs' disposal drains sit unstarted on TaskScheduler.Default — thread pool has no worker to run them (M1-by-starvation, #4847 class) #5569
Description
What is failing
During a shutdown cascade on memex-portal-deployment-777677694-p6hbn, 11 hosted sync/* hubs reached Dispose() and never progressed: RunLevel=Started, the ShutdownRequest still queued (buffer=1), drain flag latched (draining=True) but drainsInFlight=0 and drainsAwaitingScheduler=1 — the turn scheduler accepted each hub's drain and no worker ever ran it, for the whole 8 s stall budget. All 11 fired within 18 ms of each other (14:54:01.7–14:54:01.741), i.e. one teardown cascade posting every child's Dispose() from one worker.
Probable cause — M1 by thread-pool starvation (high confidence)
The counters rule out the alternatives by measurement, per Reading a Disposal Stall Verdict:
drainsInFlight=0withTurnsDequeueddelta 0 → not M2 (no drain body is blocked ahead of a handler);drainsAwaitingScheduler=1→ not M3 (a drain WAS handed to the scheduler;ScheduleDrainOne's latch-leak fix makes M3 unreachable);- these are hosted hubs, built from a fresh configuration on
TaskScheduler.Default— not a wedged/deactivated Orleans activation.
Since the #3593 fix, ScheduleDrainOne queues drains GLOBALLY (PreferFairness), so a drain that sits unstarted for 8 s means the pool genuinely had no thread to give. That is the shape issue #4847 already recorded: "queued drains receiving no worker even with PreferFairness deployed" — this incident is that residual class firing in production, with the attribution question (what occupied every worker on the pod at 14:54:01) still open.
Impact
One pod, one shutdown window, 11 hubs. Teardown of those hubs (and anything waiting on them in DisposeHostedHubs) is delayed until the pool frees a worker or an outer bound ends the process; no data loss is indicated — disposal is not forced. Secondary-path severity: a shutdown that drags, not a serving outage. If it recurs with growing counts or across pods, re-rule upward.
Where to look
- Start at
MessageService.ScheduleDrainOne/DrainLoopin MeshWeaver.Messaging andHostedHubsCollection.DisposeHubsReactive(the single-threaded cascade that posts all 11 children'sShutdownRequests). - Read the per-pod evidence around 2026-09-23T14:54:01Z on
memex-portal-deployment-777677694-p6hbn:ThreadPool.ThreadCount,PendingWorkItemCount, and what every worker was running — that attribution is the open question Hosted sync hub's disposal drain starves on an exhausted thread pool (M1-by-pool-starvation, post-#3593) #4847 left. - Known family: Hub disposal wedges before Quiescing: the action block never dequeues the buffered ShutdownRequest(Quiescing) #3593 (fixed the local-queue M1 and added the discriminating counters this verdict prints), Hosted sync hub's disposal drain starves on an exhausted thread pool (M1-by-pool-starvation, post-#3593) #4847 (global-queue drains still receiving no worker). Not a duplicate — this is the next, now-diagnosed occurrence of the residual class.
Evidence
| Fingerprint | 4520ca62dd90d213 |
| Category | MeshWeaver.Messaging.MessageHub |
| Severity | Error |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-777677694-p6hbn |
| Occurrences | 11 |
| First seen | 2026-09-23 14:54:01Z |
| Last seen | 2026-09-23 14:54:01Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Messaging.MessageHub. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-23 14:54:01Z memex-portal-deployment-777677694-p6hbn fail: MeshWeaver.Messaging.MessageHub[7316]
DISPOSAL DEADLOCK DETECTED: Hub sync/T0OVF1XSQUK6nTmHsWe3Mw made no teardown progress for 00:00:08 (last progress: sync/T0OVF1XSQUK6nTmHsWe3Mw → Started). RunLevel=Started, queue depth 1. THE PUMP IS NOT TURNING: the drain flag is latched, drainsInFlight=0, drainsAwaitingScheduler=1, and NO turn was dequeued in that window (0 dequeued in total since Dispose()). The queued work — the ShutdownRequest included — has therefore never been handed to a handler, so this stall is in THIS hub's turn scheduling and NOT in a hosted hub or a join. MECHANISM: the turn scheduler ACCEPTED 1 drain(s) and has not run them (drainsInFlight=0). The stall is in THAT SCHEDULER, not in this hub. On TaskScheduler.Default (every hosted hub — a hosted hub is built from a fresh configuration and inherits no scheduler) the drain is queued GLOBALLY (ScheduleDrainOne asks for PreferFairness since #3593), so it is visible to every worker AND to the pool's starvation detection — reading this here means the pool genuinely has no thread to give, NOT that the work is parked on the posting thread's local queue; on a ROOT GRAIN hub it is the grain's ActivationTaskScheduler (MessageHubGrain.WithTaskScheduler), where a wedged or deactivated activation parks every turn this hub will ever take. Disposal is NOT forced.
Hub sync/T0OVF1XSQUK6nTmHsWe3Mw RunLevel=Started Disposal=Pending Queue(buffer=1,deferred=0,drainsInFlight=0,openGates=0,draining=True) Registrants=6
2026-09-23 14:54:01Z memex-portal-deployment-777677694-p6hbn fail: MeshWeaver.Messaging.MessageHub[7316]
DISPOSAL DEADLOCK DETECTED: Hub sync/oLSYafozpECm3xPXB3rfKg made no teardown progress for 00:00:08 (last progress: sync/oLSYafozpECm3xPXB3rfKg → Started). RunLevel=Started, queue depth 1. THE PUMP IS NOT TURNING: the drain flag is latched, drainsInFlight=0, drainsAwaitingScheduler=1, and NO turn was dequeued in that window (0 dequeued in total since Dispose()). The queued work — the ShutdownRequest included — has therefore never been handed to a handler, so this stall is in THIS hub's turn scheduling and NOT in a hosted hub or a join. MECHANISM: the turn scheduler ACCEPTED 1 drain(s) and has not run them (drainsInFlight=0). The stall is in THAT SCHEDULER, not in this hub. On TaskScheduler.Default (every hosted hub — a hosted hub is built from a fresh configuration and inherits no scheduler) the drain is queued GLOBALLY (ScheduleDrainOne asks for PreferFairness since #3593), so it is visible to every worker AND to the pool's starvation detection — reading this here means the pool genuinely has no thread to give, NOT that the work is parked on the posting thread's local queue; on a ROOT GRAIN hub it is the grain's ActivationTaskScheduler (MessageHubGrain.WithTaskScheduler), where a wedged or deactivated activation parks every turn this hub will ever take. Disposal is NOT forced.
Hub sync/oLSYafozpECm3xPXB3rfKg RunLevel=Started Disposal=Pending Queue(buffer=1,deferred=0,drainsInFlight=0,openGates=0,draining=True) Registrants=6
2026-09-23 14:54:01Z memex-portal-deployment-777677694-p6hbn fail: MeshWeaver.Messaging.MessageHub[7316]
DISPOSAL DEADLOCK DETECTED: Hub sync/e-PK9VSCy0WAbTYtvo8Avg made no teardown progress for 00:00:08 (last progress: sync/e-PK9VSCy0WAbTYtvo8Avg → Started). RunLevel=Started, queue depth 1. THE PUMP IS NOT TURNING: the drain flag is latched, drainsInFlight=0, drainsAwaitingScheduler=1, and NO turn was dequeued in that window (0 dequeued in total since Dispose()). The queued work — the ShutdownRequest included — has therefore never been handed to a handler, so this stall is in THIS hub's turn scheduling and NOT in a hosted hub or a join. MECHANISM: the turn scheduler ACCEPTED 1 drain(s) and has not run them (drainsInFlight=0). The stall is in THAT SCHEDULER, not in this hub. On TaskScheduler.Default (every hosted hub — a hosted hub is built from a fresh configuration and inherits no scheduler) the drain is queued GLOBALLY (ScheduleDrainOne asks for PreferFairness since #3593), so it is visible to every worker AND to the pool's starvation detection — reading this here means the pool genuinely has no thread to give, NOT that the work is parked on the posting thread's local queue; on a ROOT GRAIN hub it is the grain's ActivationTaskScheduler (MessageHubGrain.WithTaskScheduler), where a wedged or deactivated activation parks every turn this hub will ever take. Disposal is NOT forced.
Hub sync/e-PK9VSCy0WAbTYtvo8Avg RunLevel=Started Disposal=Pending Queue(buffer=1,deferred=0,drainsInFlight=0,openGates=0,draining=True) Registrants=6
2026-09-23 14:54:01Z memex-portal-deployment-777677694-p6hbn fail: MeshWeaver.Messaging.MessageHub[7316]
DISPOSAL DEADLOCK DETECTED: Hub sync/iXFG5M1Ep02COKwXaU7rcA made no teardown progress for 00:00:08 (last progress: sync/iXFG5M1Ep02COKwXaU7rcA → Started). RunLevel=Started, queue depth 1. THE PUMP IS NOT TURNING: the drain flag is latched, drainsInFlight=0, drainsAwaitingScheduler=1, and NO turn was dequeued in that window (0 dequeued in total since Dispose()). The queued work — the ShutdownRequest included — has therefore never been handed to a handler, so this stall is in THIS hub's turn scheduling and NOT in a hosted hub or a join. MECHANISM: the turn scheduler ACCEPTED 1 drain(s) and has not run them (drainsInFlight=0). The stall is in THAT SCHEDULER, not in this hub. On TaskScheduler.Default (every hosted hub — a hosted hub is built from a fresh configuration and inherits no scheduler) the drain is queued GLOBALLY (ScheduleDrainOne asks for PreferFairness since #3593), so it is visible to every worker AND to the pool's starvation detection — reading this here means the pool genuinely has no thread to give, NOT that the work is parked on the posting thread's local queue; on a ROOT GRAIN hub it is the grain's ActivationTaskScheduler (MessageHubGrain.WithTaskScheduler), where a wedged or deactivated activation parks every turn this hub will ever take. Disposal is NOT forced.
Hub sync/iXFG5M1Ep02COKwXaU7rcA RunLevel=Started Disposal=Pending Queue(buffer=1,deferred=0,drainsInFlight=0,openGates=0,draining=True) Registrants=6
2026-09-23 14:54:01Z memex-portal-deployment-777677694-p6hbn fail: MeshWeaver.Messaging.MessageHub[7316]
DISPOSAL DEADLOCK DETECTED: Hub sync/RqOmPiwhOkyoPlL0vZfz8A made no teardown progress for 00:00:08 (last progress: sync/RqOmPiwhOkyoPlL0vZfz8A → Started). RunLevel=Started, queue depth 1. THE PUMP IS NOT TURNING: the drain flag is latched, drainsInFlight=0, drainsAwaitingScheduler=1, and NO turn was dequeued in that window (0 dequeued in total since Dispose()). The queued work — the ShutdownRequest included — has therefore never been handed to a handler, so this stall is in THIS hub's turn scheduling and NOT in a hosted hub or a join. MECHANISM: the turn scheduler ACCEPTED 1 drain(s) and has not run them (drainsInFlight=0). The stall is in THAT SCHEDULER, not in this hub. On TaskScheduler.Default (every hosted hub — a hosted hub is built from a fresh configuration and inherits no scheduler) the drain is queued GLOBALLY (ScheduleDrainOne asks for PreferFairness since #3593), so it is visible to every worker AND to the pool's starvation detection — reading this here means the pool genuinely has no thread to give, NOT that the work is parked on the posting thread's local queue; on a ROOT GRAIN hub it is the grain's ActivationTaskScheduler (MessageHubGrain.WithTaskScheduler), where a wedged or deactivated activation parks every turn this hub will ever take. Disposal is NOT forced.
Hub sync/RqOmPiwhOkyoPlL0vZfz8A RunLevel=Started Disposal=Pending Queue(buffer=1,deferred=0,drainsInFlight=0,openGates=0,draining=True) Registrants=6
Opened automatically from Admin/_LogIncident/4520ca62dd90d213. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 23, 2026
Duplicate of #4847: the same verdict (7316 M1-by-pool-starvation), the same hub class (hosted
sync/*), and the same counters (drainsInFlight=0,drainsAwaitingScheduler=1, 0 dequeued). The body says so itself: "this incident is that residual class firing in production". The new evidence is recorded on #4847.That evidence, briefly:
…777677694-p6hbn, 14:54:01Z) ran image3.0.0-ci.9258/9a61fed9, perOps/Status/memex-cloudfor ReplicaSet777677694, as quoted on MessageService DrainLoop throws OutOfMemoryException during hub delivery (recurrence of #4824) #5555. That image carries both earlier candidates: fix: yield synchronous hub drains to queued peers #5182 (the boundedDrainLooptrampoline) and fix(compiler): keep NodeType compiles off the silo's shared ThreadPool #5327/fix(compiler): bound compile CPU on a dedicated-thread IIoPool lane #5375 (compiles off the shared pool). Neither is sufficient.cc6zz. Cumulative GC pause went to 129 s,poolPendingto 715, and the process then died ofOutOfMemoryException(MessageService DrainLoop throws OutOfMemoryException during hub delivery (recurrence of #4824) #5555). A pod in that state stalls every pool worker, whatever schedules the drain.On why "disposal drains must not depend on the shared ThreadPool" is not the fix here: a full blocking GC stops dedicated threads too. Moving the drain to another scheduler would only change which line reports the stall. The root is the allocation burst (#5555), and that is where this family is now tracked.
Not established:
p6hbn's own[LIVENESS]heap andgcPausereadings at 14:54Z. That takes aLogsread on the control instance, which is an owner step and was not taken.