Repository navigation
RoutingGrain back-pressure: whole dispatch budget head-of-line blocked on one mesh-node-cache destination #4907
Description
What is failing
The Orleans RoutingGrain hit its 64-dispatch in-flight back-pressure threshold on all three portal pods within the same second. On two of the pods, 62 of the 64 in-flight legs were queued behind a single stream destination — cache/YORFhwiyqEyTP9ESF-NUNg — i.e. classic head-of-line blocking: one slow/stuck destination leg is holding ~the entire routing slot budget, throttling route dispatch for everything else.
Probable cause (medium confidence)
The mesh-node-cache hub for cache/YORFhwiyqEyTP9ESF-NUNg is slow or stuck: it appears as the deepest-queue destination independently on two pods, and the episode stamps show it is recurring (episode #14 on ndnxt, #5 on gx6z6) rather than a one-off drain. The third pod (s246c) logged episodes #2 and #3 on the same activation with zero queue depth and different targets — those episodes drained, confirming the routing machinery itself is healthy and the problem is destination-side. What makes that cache leg hang is not determinable from the logs — the message itself warns a slot includes the unbounded ThreadPool wait, so ThreadPool starvation on the silo cannot be ruled out; a leg that never terminates (leaked slot) also fits, since no clear/higher episode followed on the affected activations. Evidence does not support a more specific root cause.
Impact
Small burst so far (4 occurrences in ~0.4s, 2026-09-19T17:32Z, 3 pods, no recurrence since) — but the shape is concerning: cross-pod, synchronized, and one destination pinning the whole in-flight budget. If the cache leg never terminates, further dispatches on those activations will be throttled and the condition will recur. Worth a check of whether the YORFhwiyqEyTP9ESF-NUNg hub grain is still alive/making progress, and whether ThreadPool starvation was occurring on those silos at the time.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain— the dispatch/leg bookkeeping that emits this[ROUTE]line (source was not reachable from the mesh partitions I can read; the category is the starting point).- The per-destination queue around destination
cache/YORFhwiyqEyTP9ESF-NUNg— why legs to that mesh-node-cache hub do not complete. - Episode stamps
513c9499#2/#3(drained, healthy reference) vs1c7f1383#14and47069f96#5(still blocked at last sample).
Known duplicates
No prior incident or issue for this component/symptom was found in the mesh; earlier RoutingGrain log entries (warn-level, 2026-09-10/14, different fingerprints) are related machinery but not this fingerprint.
Evidence
| Fingerprint | f9798e434c66c94b |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-69956b6dbc-s246c, memex-portal-deployment-69956b6dbc-ndnxt, memex-portal-deployment-69956b6dbc-gx6z6 |
| Occurrences | 4 |
| First seen | 2026-09-19 17:32:01Z |
| Last seen | 2026-09-19 17:32:01Z |
Recent log lines
2026-09-19 17:32:01Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [513c9499#2 started 2026-09-19T17:32:01.0196661Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Store/Provision — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 17:32:01Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [1c7f1383#14 started 2026-09-19T17:32:01.0347803Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/YORFhwiyqEyTP9ESF-NUNg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 17:32:01Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [513c9499#3 started 2026-09-19T17:32:01.4008767Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target ReinsuranceDemo/SampleData — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 17:32:01Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [47069f96#5 started 2026-09-19T17:32:01.4434025Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/YORFhwiyqEyTP9ESF-NUNg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/f9798e434c66c94b. Recurrences are folded into this issue rather than opening new ones.
Activity
systemorph-com commentedon Sep 19, 2026
Still occurring: 3 more since the last update (7 total).
memex-portal-deployment-69956b6dbc-s246c,memex-portal-deployment-69956b6dbc-ndnxt,memex-portal-deployment-69956b6dbc-gx6z6Recent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 19, 2026
Consolidated onto #4908 — the
deepest 0reading of one back-pressure conditionNine open tickets report
[ROUTE] Routing back-pressurefromMeshWeaver.Hosting.Orleans.RoutingGrain. The log line supplies the discriminator itself:This ticket's samples read
stream destinations queued 0, deepest per-destination queue 0— theload / starvation reading, not head-of-line. #4908, #4798, #4905, #4907 and #4909 are all that
reading, differing only in which address happened to cross the threshold (
UWDeepfield/Cedent,Edu/LearningJourney,Store/Provision,Ifrs17/GroupOfContract,Doc/Architecture/OrleansStreamPubSubDurability) — which the line explicitly says is "the addressthat happened to cross the threshold, NOT a diagnosis." (The four tickets whose samples read
deepest 62are the head-of-line reading, and are consolidated separately onto #4904.)Why one condition produced five tickets. The message embeds the reporting activation's identity —
[4b12009b#2 started 2026-09-19T06:59:48.0117429Z]. The bot fingerprints on the normalizedmessage, and its hex-blob masking rule requires 16 or more hex characters
(
[0-9a-fA-F]{16,}), so an 8-character activation id survives normalization and eachactivation becomes its own ticket. #4908's own title already names this: "back-pressure log fans out
into per-activation incidents."
Tracking continues on #4908, which is also where the
deepest 0mechanism — dispatch slots heldacross an unbounded wait for a ThreadPool thread — is written up. Closing here as a duplicate.