Repository navigation
RoutingGrain dispatch slots saturate with non-terminating route legs (repeat of #1172/#5036) #5063
Closed as duplicate
Description
What is failing
The Orleans MeshWeaver.Hosting.Orleans.RoutingGrain on pod memex-portal-deployment-6c99c9f796-6ldfk crossed its 64-in-flight route-dispatch reporting threshold five times in a 53-second window, each crossing on a different grain activation (c24e4e43#5, 8b44740a#1, b9658e92#1, 1ccd042f#1 and #2). New activations starting at episode #1 means each freshly-activated grain immediately hit a full dispatch budget — consistent with the in-flight count never draining below half the threshold on this silo, i.e. legs not terminating (or a leaked dispatch slot) rather than a transient load spike.
Probable cause
Likely a duplicate of the known systemic RoutingGrain dispatch-slot problem (high confidence that this is not a new defect):
- This log site fans out per grain activation and the watcher folded this burst from reporter fingerprint
e4a97855ab595beb, the long-running "route dispatches in flight and not terminating" incident (13.1M occurrences since 2026-08-10 across the fleet), which was filed as issue RoutingGrain route dispatches hang at 64 in-flight when _Activity compilation stalls #1172 and later superseded by incidentAdmin/_LogIncident/3172fa7f8d939ff7/ issue RoutingGrain dispatch slots saturate: 64 in-flight legs never drain on activation a3af8356 (possible slot leak or ThreadPool starvation) #5036. - The samples carry the same signature as those siblings: mixed per-destination queue depths at the crossings (0, 39, 62, 5, 36) with
routing pool subscribingat 0–2 — backlog spread across many destinations, not head-of-line on one slow target. Activation1ccd042fre-crossed 0.24 s later as episode Test #2, meaning episode Add entire company to Open SMC Teams #1 did not drain — a slot that never released. - The specific underlying cause (route legs hung on
_Activity/compile-*calls that never return instead of failing fast) is documented on RoutingGrain route dispatches hang at 64 in-flight when _Activity compilation stalls #1172. The exact targets in this window (Indexing,Store/Install,cache/*,DeepSign/SkribbleCredential/_Activity/compile-state) differ per crossing and are explicitly not a diagnosis, but one is a compile-state activity, consistent with that pattern.
Impact
Low as an isolated burst: 10 critical lines in under a minute on one of two pods, apparently self-draining via new activations. But it is one instance of a fleet-wide pattern that has been degrading routing for over a month — see the referenced issues. Closing this as a duplicate of the systemic ticket is appropriate; the evidence here adds nothing beyond it.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain— dispatch-slot release on leg termination, and the unbounded ThreadPool wait held while a slot is taken.- Primary tickets: issue RoutingGrain route dispatches hang at 64 in-flight when _Activity compilation stalls #1172 and issue RoutingGrain dispatch slots saturate: 64 in-flight legs never drain on activation a3af8356 (possible slot leak or ThreadPool starvation) #5036 (and incident
Admin/_LogIncident/3172fa7f8d939ff7). Treat this ticket as a duplicate/close-as-dupe of that investigation. - Note: ~12 sibling back-pressure incidents were opened for the same 53-second window in this namespace (e.g.
29ad2237458a415b,2ffa882b43e2b504,351443fab3516853) — one episode, not many defects.
Evidence
| Fingerprint | fd791e377c511eef |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex |
| Pods | memex-portal-deployment-6c99c9f796-6ldfk, memex-portal-deployment-5564495d46-f8xhs |
| Occurrences | 10 |
| First seen | 2026-09-19 07:53:00Z |
| Last seen | 2026-09-19 07:53:53Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-19 07:53:24Z memex-portal-deployment-6c99c9f796-6ldfk crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [c24e4e43#5 started 2026-09-19T07:53:24.8292007Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Indexing — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 07:53:47Z memex-portal-deployment-6c99c9f796-6ldfk crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [8b44740a#1 started 2026-09-19T07:53:47.3141330Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 39, routing pool subscribing 2. Latest dispatch target Store/Install — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 07:53:51Z memex-portal-deployment-6c99c9f796-6ldfk crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [b9658e92#1 started 2026-09-19T07:53:51.0421388Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/V9iMXHCKVkurS1KMrArQ3Q — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 07:53:53Z memex-portal-deployment-6c99c9f796-6ldfk crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [1ccd042f#1 started 2026-09-19T07:53:53.7165295Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 5, routing pool subscribing 0. Latest dispatch target DeepSign/SkribbleCredential/_Activity/compile-state — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 07:53:53Z memex-portal-deployment-6c99c9f796-6ldfk crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [1ccd042f#2 started 2026-09-19T07:53:53.9600149Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 36, routing pool subscribing 0. Latest dispatch target cache/z_5Dny2pKkeS2yNS_gWx7A — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/fd791e377c511eef. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 21, 2026
Closes as duplicate of #5009 — Migrated: the mixed depths 5 / 36 / 39 / 62 in one 53-second burst, each crossing on a different, freshly-activated grain at episode #1 (§2, and the fresh-activation reading is kept at #5018 §4), plus the #1172 lineage — which is closed, while the defect is live (§7).