Repository navigation
RoutingGrain back-pressure episodes: head-of-line blocking on a cache-hub destination (62 queued legs) alongside CPU-starvation crossings #5014
Description
What is failing
RoutingGrain (MeshWeaver.Hosting.Orleans) hit its route-dispatch back-pressure threshold (64 in flight) on two pods of the memex-portal deployment within the same second. The two samples show different underlying conditions, so treat this ticket as covering both shapes at this log site:
- Pure load (pod
...-zzvfq): 64 in flight, 0 destinations queued, deepest per-destination queue 0 — every leg was independently moving; the silo was simply CPU/ThreadPool-starved (the log's own guidance: a slot is held through the unbounded wait for a ThreadPool thread). - Genuine head-of-line blocking (pod
...-74xjv): 64 in flight with a deepest per-destination queue of 62 behind a single destination — a cache hub (cache/Lmzw6ArfrECTSTdqx1hFTg). 62 legs queued behind one leg is exactly the "deliveries waiting on a delivery" signature the alert redesign (see Doc/WhatsNew 2026-08-12) said is the real blockage signal.
Probable cause
- Shape 2 (medium-high confidence): a single route leg to a cache-hub destination stalled, and 62 subsequent dispatches to that destination queued behind it. Candidate causes: the destination hub is stuck or slow to drain, or a leg failed to terminate without releasing its slot (the slot-leak failure mode the same doc says was fixed for shutdown-cancellation — a different leak path may remain). The episode stamp
b5b0cd54#1was never followed by a higher episode or a clear line in this window, consistent with a leg that never terminated. - Shape 1 (high confidence, and not a defect on its own): threshold crossing under CPU starvation — expected under load, the alert is informational here.
Impact
Small in isolation: 2 occurrences over ~1s on 2 pods. However, the Admin/_LogIncident space contains dozens of sibling back-pressure incidents from the same component over the last two days, plus separate OutOfMemoryException route-dispatch incidents — this is a recurring condition under load, not a one-off. When it recurs with the shape-2 signature, deliveries to the affected destination are delayed en masse; shape 1 only indicates fleet-wide CPU pressure.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain— the dispatch bookkeeping: slot acquisition/release on leg termination, and the per-destination ordered queues (shape 2 says the head leg on a destination must not be able to block 62 followers indefinitely without a timeout).- The cache-hub destination
cache/Lmzw6Arfrfr…(cache/Lmzw6ArfrECTSTdqx1hFTg) on pod...-74xjv— check whether that hub was stuck at 2026-09-20T21:56:14Z. - Design context: Doc/WhatsNew/2026-08-12-routing-alerts-name-what-they-actually-saw.
Possible duplicate / related
Many sibling back-pressure incidents exist for this component (e.g. Admin/_LogIncident/08ebdf437a2311bf, e6ab8b9a095963a9, ae26d036ed54dc47, all from the same 21:56 window; earlier ones on 2026-09-19). If a tracking issue already exists for routing back-pressure episodes, this can be folded into it — the distinct contribution here is the 62-deep single-destination head-of-line queue, which is a real blockage, not load.
Evidence
| Fingerprint | 75365ef07a58ce95 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex |
| Pods | memex-portal-deployment-6b7c4f94db-zzvfq, memex-portal-deployment-5f69f6b6bd-74xjv |
| Occurrences | 2 |
| First seen | 2026-09-20 21:56:13Z |
| Last seen | 2026-09-20 21:56:14Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-20 21:56:13Z memex-portal-deployment-6b7c4f94db-zzvfq crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [4b65d49f#1 started 2026-09-20T21:56:13.9094547Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Admin/_LogIncident/2b608cd1880c7d52/_Thread/triage-the-log-incident-at-admin-loginci-d818/6597f971 — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 21:56:14Z memex-portal-deployment-5f69f6b6bd-74xjv crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [b5b0cd54#1 started 2026-09-20T21:56:14.7790572Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/Lmzw6ArfrECTSTdqx1hFTg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/75365ef07a58ce95. Recurrences are folded into this issue rather than opening new ones.
Activity
systemorph-com commentedon Sep 20, 2026
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-6b7c4f94db-zzvfq,memex-portal-deployment-5f69f6b6bd-74xjvRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 21, 2026
Closes as duplicate of #5009 — Migrated: destination
cache/Lmzw6ArfrECTSTdqx1hFTg(§1) and the episodeb5b0cd54#1that was never followed by a higher episode or a clear line — the second sample in the set that does not exclude a leaked slot (§3).fix(routing): the saturation report prints the gauge that can see the…