Repository navigation
RoutingGrain dispatch pool saturates under routine load: 61+ legs wait for pool slot with empty channel queues #5436
Description
What is failing
The Orleans RoutingGrain in the portal is repeatedly hitting its dispatch back-pressure threshold (64 in flight) on both pods of the memex-portal deployment, with ~61–62 of those legs parked waiting for a pool slot while the ordered-channel queues are empty (0 queued, deepest 0) and subscribing reads ~0–1. In other words: the routing pool's slot cap is the bottleneck, not any downstream destination, stream, or stuck leg.
Probable cause — medium confidence
Per the log line's own diagnostic contract (and issues #5009 / #5018 it cites): every leg is young (oldest 12–29 ms), channels queued 0, and the wait is in the pre-subscribe pool-slot acquisition. That shape reads as routine burst load hitting a fixed pool cap — a capacity/tuning gap (cap too low for normal dispatch fan-out, or ThreadPool starvation on CPU-pressed silos holding slots through the unbounded pre-timeout wait) rather than a slot leak or head-of-line blockage. No leg in these samples is minutes old and no episode shows a never-draining in-flight count, so a leak is not supported by this evidence. The recurrence of the same shape across many episodes on multiple pods over several days suggests the threshold/pool cap is simply mis-sized for steady-state traffic.
Impact
Seven occurrences in a ~3-second window across 2 pods for this episode, but the same log site has produced dozens of sibling incidents over the past days. Dispatches are briefly delayed (back-pressure engages, then drains — legs never age past tens of ms), so user impact is latency jitter, not loss. No user-visible failure is demonstrated.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain — the dispatch pool sizing / slot-acquisition logic and the reporting threshold (64). Consider raising the pool cap, bounding or instrumenting the pre-subscribe ThreadPool wait, or re-tuning the threshold so routine bursts stop paging on-call. This incident is likely a near-duplicate of the already-known routing back-pressure cluster (cf. issues #5009 and #5018 referenced in the message text) — close as dup if a ticket for pool-cap saturation already exists.
Evidence
| Fingerprint | 0c95530681c59563 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex |
| Pods | memex-portal-deployment-5c99d79997-xs87x, memex-portal-deployment-5c99d79997-7qdb4 |
| Occurrences | 7 |
| First seen | 2026-09-23 03:42:58Z |
| Last seen | 2026-09-23 03:43:01Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-23 03:42:58Z memex-portal-deployment-5c99d79997-xs87x crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [8113be95#11 started 2026-09-23T03:42:58.3754154Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 0, waiting for a pool slot 61; oldest leg in flight 16 ms — dispatch → Doc/WhatsNew/2026-08-16-content-upload-empty-render-race (delivery 2X_KvyL4MEyGh7mk7N7-Wg). Latest dispatch target Doc/Architecture/SharedRuleBlocks — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sa…[truncated]
2026-09-23 03:43:01Z memex-portal-deployment-5c99d79997-7qdb4 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [99189ff2#3 started 2026-09-23T03:43:01.0480707Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 0, waiting for a pool slot 61; oldest leg in flight 29 ms — dispatch → Doc/WhatsNew/2026-08-09-token-checks-no-longer-run-on-the-router (delivery Miq90pQTr0Sq1y0rZGHG0Q). Latest dispatch target Doc/WhatsNew/2026-08-25-temporary-problems-stop-looking-permanent — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND…[truncated]
2026-09-23 03:43:01Z memex-portal-deployment-5c99d79997-7qdb4 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [99189ff2#4 started 2026-09-23T03:43:01.2626724Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 0, waiting for a pool slot 61; oldest leg in flight 12 ms — dispatch → Doc/WhatsNew/2026-08-10-a-rebuilt-node-type-no-longer-breaks-open-pages (delivery rx2NLwUXmEye9LYlW9AU4A). Latest dispatch target Doc/Architecture/SearchCoverageAndRefusal — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anyt…[truncated]
2026-09-23 03:43:01Z memex-portal-deployment-5c99d79997-7qdb4 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [99189ff2#5 started 2026-09-23T03:43:01.2991365Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 0, waiting for a pool slot 62; oldest leg in flight 20 ms — dispatch → ThinkInStreams (delivery hDkkWZHr4Ea4RabiMyMemg). Latest dispatch target Doc/WhatsNew/2026-08-07-red-log-ticketing — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so like the …[truncated]
2026-09-23 03:43:01Z memex-portal-deployment-5c99d79997-7qdb4 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [99189ff2#6 started 2026-09-23T03:43:01.3215323Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 1, waiting for a pool slot 62; oldest leg in flight 24 ms — dispatch → Doc/WhatsNew/2026-08-25-mesh-wide-paging-works (delivery --HEEl-DYU-0AwUoHMOrIw). Latest dispatch target Doc/WhatsNew/2026-09-01-a-module-brings-its-own-dependencies-again — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anyt…[truncated]
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/0c95530681c59563. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 23, 2026
Duplicate of #5118.
Build: pod(s)
memex-portal-deployment-5c99d79997-xs87x,memex-portal-deployment-5c99d79997-7qdb4in namespacememex. ReplicaSet5c99d79997runs3da3423c/3.0.0-ci.9193, which does not contain the compile-off-ThreadPool fix (#5327/#5375) (read fromOps/Status/memex). memex has not rolled yet, so this is the build that #5118's post-roll verification still has to replace.Window 2026-09-23 03:42:58Z → 2026-09-23 03:43:01Z, 7 occurrence(s), fingerprint
0c95530681c59563.8113be95#1199189ff2#399189ff2#499189ff2#599189ff2#6This issue's own discriminator: thread shortage (T): waiting for a pool slot high while subscribing ≤ 1. The oldest leg is at most 29 ms, so no leg is minutes old and no slot leaked. Episodes keep advancing on the same activation (
99189ff2), so each episode drained. It is the same log site and the same reading, sampled on the build without the fix. The cause, the fix, and the verification after memex rolls are recorded on #5118.