Repository navigation
RoutingGrain reports Critical back-pressure during transient ThreadPool starvation; legs hold slots while queuing for a thread #4843
Description
What is failing
MeshWeaver.Hosting.Orleans.RoutingGrain (category MeshWeaver.Hosting.Orleans) raised the routing back-pressure alarm at Critical severity across all 3 memex-portal pods in a single ~2-minute window. All in-flight legs (64 = threshold) were past the queueing stage — stream destinations queued 0, deepest per-destination queue 0 — yet the routing pool was saturated. Per the log's own semantics, this pattern means the silo's threads, not the destinations, were the bottleneck: each dispatch holds a routing slot through the unbounded wait for a ThreadPool thread, so a thread-starved silo trips the alarm with nothing actually stuck.
Probable cause
Medium confidence: ThreadPool starvation on the portal silos during a transient burst — the dispatch targets in the samples (*_Install/ICloud, *_Install/OpenCode, Store/Coupon, Store/Purchase, an Essentials activity) look like a mass start-up/install burst that momentarily needed more concurrent legs than the pool had threads. Supporting evidence: zero queued destinations (no head-of-line blocking), episode numbers climbing on the same activation (#6→#7→#8 on 74d0b3b5), meaning each back-pressure episode drained fully before re-crossing — no leaked slots, no blocked legs. An alternative — a genuinely leaked slot — is ruled out for the sampled activation by the rising episode stamps; whether other activations cleared cannot be confirmed from the 5 samples, but the burst ended on its own at 07:52 and did not recur.
Impact
Bounded and transient: 11 occurrences over 102 seconds, all pods, self-resolved, nothing in the 4 hours since. No evidence of stuck or dropped routes. User-facing impact likely brief dispatch latency during the burst, if any.
Where to look
- The slot-lifecycle in
MeshWeaver.Hosting.Orleans.RoutingGrainroute dispatch: the slot is held from dispatch through the ThreadPool wait before leg timeouts start. Consider bounding that wait (or starting the leg timeout before the thread is acquired) so a CPU-starved silo doesn't consume the entire budget. - The reporting threshold of 64 concurrent dispatches is low enough that an ordinary burst trips a Critical; if this is expected load shape, revisit the threshold/severity mapping.
- Check silo ThreadPool starvation metrics for 07:50–07:52 on 2026-09-19 to confirm thread-hunger vs. a real routing defect.
No existing incident or issue for this component was found in the mesh.
Evidence
| Fingerprint | b8b9d012e5fb9908 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-7978987466-q6m89, memex-portal-deployment-7978987466-sq6sq, memex-portal-deployment-7978987466-kmrpk |
| Occurrences | 11 |
| First seen | 2026-09-19 07:50:34Z |
| Last seen | 2026-09-19 07:52:16Z |
Recent log lines
2026-09-19 07:51:42Z memex-portal-deployment-7978987466-q6m89 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [74d0b3b5#6 started 2026-09-19T07:51:42.8167706Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target bari/_Install/ICloud — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 07:51:59Z memex-portal-deployment-7978987466-sq6sq crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [7fc42f2a#1 started 2026-09-19T07:51:59.0122387Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 1. Latest dispatch target Store/Coupon — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 07:51:59Z memex-portal-deployment-7978987466-kmrpk crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [431727b6#5 started 2026-09-19T07:51:59.5535013Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Store/Purchase — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 07:52:01Z memex-portal-deployment-7978987466-q6m89 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [74d0b3b5#7 started 2026-09-19T07:52:01.6590791Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target baisakhi.das/_Install/OpenCode — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 07:52:16Z memex-portal-deployment-7978987466-q6m89 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [74d0b3b5#8 started 2026-09-19T07:52:16.4699432Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Essentials/_Activity/83c4d789 — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/b8b9d012e5fb9908. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 19, 2026
Closing as a duplicate of #4798.
This issue and 46 others were filed today by one log site —
MeshWeaver.Hosting.Orleans.RoutingGrain's back-pressure line — because the Orleans activation id in that line was not masked before the fingerprint was taken.LogLineParser.HexBlobrequired sixteen hex characters; an activation id is eight, so it fell through to the number masker, which stripped its digits and left its letters behind as literal text. An activation id changes on every grain recycle and every pod roll, and memex-cloud rolled repeatedly today, so each activation minted a new fingerprint, a new incident and a new issue.These are not 47 defects. Under correct masking all 47 normalize to one message and would have been one incident. Verified two of them against the control instance's own records:
Admin/_LogIncident/5b6a59648c91fd85andeb29857024b3f0eddiffer only incfb5d6vsc0cc8— the dispatch target is already{path}in both, and every count is already{n}.Fixed in Systemorph/MeshWeaver.Plugins#2165 (merged), which masks a run of ≥ 8 hex characters carrying both a digit and an a–f letter. The fault itself — routing back-pressure at the 64-dispatch threshold — remains open and is tracked on #4798, which carries the longest occurrence history.
Reopen if you believe this one is a distinct fault rather than another activation of the same line.