Repository navigation
RoutingGrain head-of-line blockage: ~63 route legs queue behind a single non-terminating leg to a cache destination #4995
Description
What is failing
The Orleans RoutingGrain route dispatch back-pressure guard is saturating: 64 dispatches in flight (at the reporting threshold) repeatedly within a 20-second window on both portal pods. Crucially, the later crossings on activation b39b1032 show 1 stream destination queued with a deepest per-destination queue of 62 — i.e. ~63 route legs are queued behind a single leg in a head-of-line blockage, not spread across destinations as ordinary load. The blocked destination is the cache node cache/pY9MmqY9YECUi-OLU-u1pA in both late samples.
Probable cause
Medium-high confidence: one route leg to that cache destination is not terminating, and per-destination head-of-line queuing backs the remaining legs up behind it. Evidence: episodes #4→#7 rose on the same activation (so each episode did drain — this is not a leaked-slot episode per the alert's own reading rule), but the backlog refilled to the threshold within seconds and the late samples show near-all in-flight legs queued on a single destination with the same target address one second apart. The first two crossings and the crossing on the other pod had zero queuing (pure load shape), so the burst likely began as load and the blockage to cache/pY9MmqY9YECUi-OLU-u1pA formed on top of it. An alternative — ThreadPool starvation on the silo — does not explain a 62-deep queue on exactly one destination.
Impact
Limited so far: 5 occurrences in ~20 seconds on the two memex-portal-deployment pods; episodes drained (rising episode numbers on the same activation) so dispatch kept flowing. However, a stuck leg that persists would leak-look like the shutdown-cancellation slot-leak fixed in the 2026-08-12 routing-alerts rework — if neither a clear nor a higher episode ever follows, in-flight never falls below half the threshold. Worth checking whether deliveries to that cache address are actually completing today.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain — the dispatch-slot accounting and per-destination queue: why a leg to cache/pY9MmqY9YECUi-OLU-u1pA holds its slot without terminating, and whether leg timeouts bound the wait. Context: Doc/WhatsNew/2026-08-12-routing-alerts-name-what-they-actually-saw documents a previously fixed slot-leak at shutdown in this same component.
Evidence
| Fingerprint | 8c8a5638c4fddae7 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-69956b6dbc-s246c, memex-portal-deployment-69956b6dbc-t88pn |
| Occurrences | 5 |
| First seen | 2026-09-20 20:37:47Z |
| Last seen | 2026-09-20 20:38:06Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-20 20:37:47Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [b39b1032#4 started 2026-09-20T20:37:47.4792093Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target TaskManagement/TeamMember — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 20:37:47Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [b39b1032#5 started 2026-09-20T20:37:47.5045962Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Planning/Variance — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 20:38:05Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [b39b1032#6 started 2026-09-20T20:38:05.6176729Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/pY9MmqY9YECUi-OLU-u1pA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 20:38:06Z memex-portal-deployment-69956b6dbc-t88pn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [2b2c7c6c#1 started 2026-09-20T20:38:06.4428458Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Ifrs17/DataNodeState — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 20:38:06Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [b39b1032#7 started 2026-09-20T20:38:06.5173903Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/pY9MmqY9YECUi-OLU-u1pA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/8c8a5638c4fddae7. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 21, 2026
Closes as duplicate of #5009 — Migrated: destination
cache/pY9MmqY9YECUi-OLU-u1pA(§1). Otherwise the same 62-deep shape as the canonical.