Repository navigation
RoutingGrain dispatch pool repeatedly saturates under burst load, triggering back-pressure alerts across the portal silo #5409
Description
What is failing
The routing grain's dispatch pool on the memex-portal silo saturates under burst load: 64 route dispatches in flight hit the reporting threshold, with 59–61 legs waiting for a pool slot while subscribing reads 0. Every episode drained (episode stamps climb #4 → #5 → #6 on the same activation) and the oldest leg was ~20 ms — load, not a leaked slot or head-of-line block.
Probable cause
Confidence: moderate. The gauges indicate legs were queuing for a pool slot before their subscribe prologue — i.e., the dispatch pool (or the ThreadPool serving it, per the gauge semantics documented for issue #5018) was briefly exhausted during a dispatch burst. The burst self-healed within seconds each time, so nothing was stuck; but this fingerprint is one of dozens of identical back-pressure incidents opened across pods in the past week, which points at the routing pool capacity being too small for current dispatch volume rather than any single-burst fluke.
Impact
Transient latency on route dispatches during bursts on a single portal pod; 3 occurrences over ~2.5 minutes, self-draining, no lost or stuck deliveries. Nuisance today, but the recurrence rate suggests it will worsen as load grows.
Where to look
The routing dispatch pool sizing/semaphore logic in MeshWeaver.Hosting.Orleans.RoutingGrain — the dispatch-slot acquisition path and the threshold of 64 in-flight legs. Also worth a holistic look at the back-pressure incident family (this is a recurring shape across many activations/pods since 2026-09-19) to decide whether pool capacity or the reporting threshold needs tuning.
Evidence
| Fingerprint | 0bec1f673f113f71 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-857546649-vf2q6 |
| Occurrences | 3 |
| First seen | 2026-09-22 21:24:17Z |
| Last seen | 2026-09-22 21:26:40Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-22 21:24:17Z memex-portal-deployment-857546649-vf2q6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [eb5398a8#4 started 2026-09-22T21:24:17.1174977Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 0, waiting for a pool slot 61; oldest leg in flight 25 ms — dispatch → Doc/WhatsNew/2026-07-22-kernel-memory-unload (delivery NDk-pytP_kq8XnJuIICN3A). Latest dispatch target HomeAssistant/Home — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so li…[truncated]
2026-09-22 21:24:17Z memex-portal-deployment-857546649-vf2q6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [eb5398a8#5 started 2026-09-22T21:24:17.5266493Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 0, waiting for a pool slot 25; oldest leg in flight 19 ms — dispatch → Doc/WhatsNew/2026-08-06-side-menu-order (delivery OMXfBKZETki1OG2rhRjERQ). Latest dispatch target Ifrs17/CreditRiskRating — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so li…[truncated]
2026-09-22 21:26:40Z memex-portal-deployment-857546649-vf2q6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [eb5398a8#6 started 2026-09-22T21:26:40.0963196Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 0, waiting for a pool slot 59; oldest leg in flight 20 ms — dispatch → ILS/Transaction (delivery K4tAWEkydEWLjP5osrG7BQ). Latest dispatch target Store/Core — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly a…[truncated]
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/0bec1f673f113f71. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 23, 2026
Duplicate of #5118.
Build: pod(s)
memex-portal-deployment-857546649-vf2q6. ReplicaSet857546649runs3da3423c/3.0.0-ci.9193, which does not contain the compile-off-ThreadPool fix (#5327/#5375) (read fromOps/Status/memex-cloudversion 3130).Window 2026-09-22 21:24:17Z → 2026-09-22 21:26:40Z, 3 occurrence(s), fingerprint
0bec1f673f113f71.eb5398a8#4eb5398a8#5eb5398a8#6This issue's own discriminator: thread shortage (T): waiting for a pool slot high while subscribing ≤ 1. The oldest leg is at most 25 ms, so no leg is minutes old and no slot leaked. Episodes keep advancing on the same activation (
eb5398a8), so each episode drained. It is the same log site and the same reading, sampled on the build without the fix. The cause, the fix, and the verification after memex rolls are recorded on #5118.