Repository navigation
RoutingGrain back-pressure crossing on memex-portal: 63 legs briefly waiting on routing-pool slots (thread starvation signature, no stuck leg) #5425
Description
What is failing: The RoutingGrain on the memex-portal pod crossed its back-pressure reporting threshold: 64 route dispatches in flight, with 63 legs waiting for a routing-pool slot and 0 anywhere subscribed. No channel was head-of-line blocked (all channel queues 0) and the oldest leg was 15 ms old — nothing was stuck or leaked.
Probable cause (medium-high confidence): A momentary ThreadPool thread starvation / load spike on the silo, per the log line's own gauges — waiting for a pool slot high while subscribing reads 0 is the documented signature of legs queued before their subscribe prologue, and an all-young leg population (oldest 15 ms) means the episode drained immediately rather than a slot leaking. The incident is therefore most likely transient load the silo absorbed, not a defect in routing itself. If these recur on this pod, look at CPU sizing of memex-portal-deployment or whatever else was consuming threads at 01:10 UTC.
Impact: One occurrence on one pod (memex-portal-deployment-5c99d79997-xs87x); deliveries were delayed by at most tens of milliseconds. No user-visible failure is evidenced.
Where to look: MeshWeaver.Hosting.Orleans.RoutingGrain ([ROUTE] log site) and the routing pool's slot acquisition / pre-subscribe wait. Related prior art: this alert family is known to mint noisy Critical tickets per episode stamp — see rbuergi/Feedback/route-backpressure-fingerprint-splits-20260921-2236 — and recent Ops/Logs entries show comparable episodes clearing in ~60 ms. Consider deduplicating/suppressing short-lived crossings of this fingerprint.
Evidence
| Fingerprint | 985f04359b7108a1 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex |
| Pods | memex-portal-deployment-5c99d79997-xs87x |
| Occurrences | 1 |
| First seen | 2026-09-23 01:10:58Z |
| Last seen | 2026-09-23 01:10:58Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-23 01:10:58Z memex-portal-deployment-5c99d79997-xs87x crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [b96b00b1#5 started 2026-09-23T01:10:58.1193873Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 0, waiting for a pool slot 63; oldest leg in flight 15 ms — dispatch → Doc/WhatsNew/2026-09-19-the-local-test-loop-nobody-could-start (delivery y8cMT-ge0kCWU1aJFna9iQ). Latest dispatch target Doc/WhatsNew/2026-09-19-a-build-handoff-that-reported-success-and-arrived-nowhere — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below …[truncated]
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/985f04359b7108a1. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 23, 2026
Duplicate of #5118.
Build: pod(s)
memex-portal-deployment-5c99d79997-xs87xin namespacememex. ReplicaSet5c99d79997runs3da3423c/3.0.0-ci.9193, which does not contain the compile-off-ThreadPool fix (#5327/#5375) (read fromOps/Status/memex). memex has not rolled yet, so this is the build that #5118's post-roll verification still has to replace.Window 2026-09-23 01:10:58Z → 2026-09-23 01:10:58Z, 1 occurrence(s), fingerprint
985f04359b7108a1.b96b00b1#5This issue's own discriminator: thread shortage (T): waiting for a pool slot high while subscribing ≤ 1. The oldest leg is at most 15 ms, so no leg is minutes old and no slot leaked. It is the same log site and the same reading, sampled on the build without the fix. The cause, the fix, and the verification after memex rolls are recorded on #5118.