Repository navigation
RoutingGrain: single stalled cache destination head-of-line blocks ~75% of routing dispatch slots (recurring) #5132
Description
What is failing
The Orleans RoutingGrain (MeshWeaver.Hosting) crossed its route-dispatch back-pressure threshold: 64 dispatch legs in flight, with 48 of them queued behind a single cache/… stream destination — a head-of-line pattern where one stalled destination pins ~75% of the routing grain's entire in-flight budget.
Probable cause
A single cache-destination leg stalled (slow/stalled consumer, or an unbounded wait for a ThreadPool thread before its own timeouts begin — the log site itself flags this) and per-destination FIFO ordering blocked the rest of that destination's traffic behind it. Medium confidence for this single episode: no follow-up log line with a higher episode on activation 71bdb43f is in evidence, so a leaked slot (leg never terminated) or a transient episode drained normally cannot be distinguished here. However, the pattern is corroborated by dozens of sibling incidents at this same log site across 2026-09-19 → 09-21 (e.g. Admin/_LogIncident/0fc7665eaf6aae43, 155d7ffcac9123d3, 03e5e31ad393af55, …), each showing the deepest queue concentrated on a cache destination — head-of-line blocking on cache legs is the systemic defect, not a one-off.
Impact
This episode itself: 2 log lines, one pod, 09:44 UTC today — minor. But the recurring family (many distinct activation ids per day for 3+ days on the memex portal deployment) means routing throughput is repeatedly throttled for all tenants routed through the grain, and each episode mints a new incident, drowning the triage queue.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain — the dispatch-slot bookkeeping (slot held from dispatch until leg termination, including the unbounded ThreadPool wait) and the per-destination queue. Candidate fixes:
- Bound the wait for a ThreadPool thread (or count timeouts from dispatch, not from thread start) so a starved silo cannot hold slots indefinitely.
- Per-destination queue cap / leg timeout so one stalled
cachedestination cannot consume the global in-flight budget. - Ops fix: mask the activation id and episode number out of the incident fingerprint for this log site, so episodes fold into one trackable incident instead of dozens.
Evidence
| Fingerprint | f15421809195b444 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex |
| Pods | memex-portal-deployment-5998d5fddf-75t7b |
| Occurrences | 2 |
| First seen | 2026-09-21 09:44:24Z |
| Last seen | 2026-09-21 09:44:24Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-21 09:44:24Z memex-portal-deployment-5998d5fddf-75t7b crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [71bdb43f#1 started 2026-09-21T09:44:24.4810317Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 2, deepest per-destination queue 48, routing pool subscribing 0. Latest dispatch target cache/wpWb1fSJPUacoVxY6LnltQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 09:44:24Z memex-portal-deployment-5998d5fddf-75t7b crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [71bdb43f#1 started 2026-09-21T09:44:24.4810317Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 2, deepest per-destination queue 48, routing pool subscribing 0. Latest dispatch target cache/wpWb1fSJPUacoVxY6LnltQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/f15421809195b444. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 21, 2026
Closes as duplicate of #5009.
Reason: same defect —
RoutingGraindispatch slots saturate at the 64-in-flight reporting threshold with legs head-of-line blocked behind a single destination. Grouped on the discriminator the log line itself defines, not on the title: a deepest per-destination queue of 1 or more means legs are blocked behind a leg. This ticket measured deepest per-destination queue 48 of a 64-slot budget over 2 queued destination(s) — so the great majority of the budget was held behind one leg, which is #5009's subject and not the empty-queue load shape tracked separately at #5018.Evidence: namespace
memex, 2 occurrence(s), 2026-09-21 09:44:24Z → 2026-09-21 09:44:24Z, episode stamp(s)71bdb43f#1, latest dispatch target(s)cache/wpWb1fSJPUacoVxY6LnltQ. Fingerprintf15421809195b444.Unique evidence — migrated to #5009: same blocked destination
cache/wpWb1fSJPUacoVxY6LnltQ; ~75% of the dispatch budget consumed, and the reading recurred.