Repository navigation
RoutingGrain dispatch pool saturates (64 in-flight cap) under tenant fan-out; legs queue pre-subscribe — recurrence of #5368 #5423
Description
What is failing
The RoutingGrain route-dispatch pool (64 slots) on the memex-cloud portal saturates: all 64 dispatch legs in flight, ~57–63 of them queued waiting for a pool slot while subscribing reads 0–1. By the log line's own diagnostic contract that means legs are blocked before entering their subscribe prologue — the pool's worker capacity cannot absorb the dispatch rate. Dispatch targets span many unrelated tenants/users, so this is fan-out load, not a single poisoned destination.
Probable cause
High confidence (~80%) that this is load, not a slot leak. Evidence:
oldest leg in flightis 3 ms–7.5 s across all samples — no leg is minutes old, so no slot never came back.- Channels queued 0 / deepest 0 — no head-of-line blocking within any stream; the bottleneck is purely pool capacity.
- Episode
847c7c68Add entire company to Open SMC Teams #1→Clarify license options with ag grid #4 advances, then a new activation iddef8db1copens Add entire company to Open SMC Teams #1 at 23:11:13 — the episode drained (or the grain was recycled) and re-crossed. A later line with a higher episode or new activation means the backlog clears; only the sustained absence of either would indicate a leak, and we do not see that.
Likely mechanism: the 64-slot dispatch cap (and/or ThreadPool starvation on the pod — a leg holds its slot through the unbounded wait for a ThreadPool thread before its own timeouts start) is too small for this silo's tenant fan-out.
Impact
30 occurrences in a ~3-minute window on one pod, episodes draining each time — transient dispatch latency on routed stream deliveries, no evidence of stuck or lost legs. Users may see delayed stream updates during bursts. Not release-blocking on this evidence.
Likely duplicate of #5368
This is the same condition as issue #5368 (incident Admin/_LogIncident/0674869797fb855a, filed ~4.5 h earlier): same category, same gauge pattern (subscribing ~0, waiting-for-slot ~60, channels 0/0, legs all young, episodes draining), different pod. Many sibling back-pressure incidents with distinct episode stamps were minted today — the fingerprint splits per episode stamp. A human should consider closing this as a duplicate of #5368; the underlying ticket-spam issue is already tracked in rbuergi/Feedback/route-backpressure-fingerprint-splits-20260921-2236.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain— dispatch pool sizing (threshold 64) and the slot-acquisition path before the subscribe prologue; consider raising the cap or bounding the pre-subscribe wait.- ThreadPool starvation metrics on
memex-portal-deployment-857546649-vf2q6for the 23:08–23:11 UTC window.
Evidence
| Fingerprint | 45eb25d06c0ee39f |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-857546649-vf2q6 |
| Occurrences | 30 |
| First seen | 2026-09-22 23:08:03Z |
| Last seen | 2026-09-22 23:11:13Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-22 23:10:44Z memex-portal-deployment-857546649-vf2q6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [847c7c68#1 started 2026-09-22T23:10:44.4255113Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 0, waiting for a pool slot 61; oldest leg in flight 3882 ms — dispatch → juergen.schwaermer/_Install/DefaultViews (delivery TlS2KAlIikiQF4cwyT4dYw). Latest dispatch target Google/_Access/jacky.enciso_Access — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE C…[truncated]
2026-09-22 23:10:51Z memex-portal-deployment-857546649-vf2q6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [847c7c68#2 started 2026-09-22T23:10:51.9531397Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 0, waiting for a pool slot 60; oldest leg in flight 7514 ms — dispatch → GraphViews (delivery JEMYS2Qu0EOSKfOL8WjUZQ). Latest dispatch target OpenStreetMap/_Entitlements/roswitha.escher — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so like the …[truncated]
2026-09-22 23:10:55Z memex-portal-deployment-857546649-vf2q6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [847c7c68#3 started 2026-09-22T23:10:55.5736725Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 0, waiting for a pool slot 62; oldest leg in flight 3539 ms — dispatch → RolePlay/_Access/kalpesh.patel_Access (delivery 0qVqxTbjBkuCvY0WqPXP6A). Latest dispatch target Publish/_Policy — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so like the i…[truncated]
2026-09-22 23:10:55Z memex-portal-deployment-857546649-vf2q6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [847c7c68#4 started 2026-09-22T23:10:55.6137106Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 1, waiting for a pool slot 62; oldest leg in flight 4 ms — dispatch → denise.munari/_Install/LearningRoadmap (delivery hW-Qe64ICkmD1HigCRuz4A). Latest dispatch target Google/_Entitlements/rene.hug — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, s…[truncated]
2026-09-22 23:11:13Z memex-portal-deployment-857546649-vf2q6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [def8db1c#1 started 2026-09-22T23:11:13.0061357Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 1, waiting for a pool slot 57; oldest leg in flight 3222 ms — dispatch → roger.sas2026/_Install/Voice (delivery UtPZI3Ebek6iDorZVOKiWg). Latest dispatch target Pricing/SimulationSettings/_Activity/compile-state — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT T…[truncated]
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/45eb25d06c0ee39f. Recurrences are folded into this issue rather than opening new ones.
Activity
systemorph-com commentedon Sep 23, 2026
Still occurring: 0 more since the last update (30 total).
memex-portal-deployment-857546649-vf2q6Recent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 23, 2026
Duplicate of #5118.
Build: pod(s)
memex-portal-deployment-857546649-vf2q6. ReplicaSet857546649runs3da3423c/3.0.0-ci.9193, which does not contain the compile-off-ThreadPool fix (#5327/#5375) (read fromOps/Status/memex-cloudversion 3130).Window 2026-09-22 23:08:03Z → 2026-09-22 23:11:13Z, 30 occurrence(s), fingerprint
45eb25d06c0ee39f.847c7c68#1847c7c68#2847c7c68#3847c7c68#4def8db1c#1This issue's own discriminator: thread shortage (T): waiting for a pool slot high while subscribing ≤ 1. The oldest leg is at most 7514 ms, so no leg is minutes old and no slot leaked. Episodes keep advancing on the same activation (
847c7c68), so each episode drained. It is the same log site and the same reading, sampled on the build without the fix. The cause, the fix, and the verification after memex rolls are recorded on #5118.