Repository navigation
RoutingGrain back-pressure episode on memex pod: 36-deep destination queue (duplicate of fleet-wide #1172) #5139
Description
What is failing
MeshWeaver.Hosting.Orleans.RoutingGrain crossed its 64 in-flight dispatch threshold on the memex pod. The episode (c0d85529#1) shows 64 dispatches in flight with 4 stream destinations queued and a deepest per-destination queue of 36, with the crossing target portal/nodeops-… — the classic head-of-line shape: over half the in-flight slots are held behind legs queued on a small number of slow destinations.
Probable cause (moderate confidence)
Same mechanism as the fleet-wide RoutingGrain saturation already filed: a dispatch slot is held from dispatch until the leg terminates, including the unbounded wait for a ThreadPool thread, and legs queued behind one slow destination accumulate in a per-destination queue (36 deep here). No second episode or drain line followed on this activation in the captured window, so whether this episode cleared or the slot leaked cannot be determined from the incident evidence alone — treat as the known back-pressure condition, not a new defect.
Impact
Minimal for this instance: a single burst of 2 occurrences on one pod at one timestamp. The systemic impact is tracked under #1172, which shows 13M+ occurrences across ~500 pods.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain— dispatch-slot accounting and per-destination queue behaviour (see RoutingGrain route dispatches hang at 64 in-flight when _Activity compilation stalls #1172).- Check the episode stamp
c0d85529in raw logs for a later/higher episode to confirm the backlog drained.
Already known
Almost certainly a duplicate of #1172 — same log category, same message shape, same head-of-line signature on cache/portal destinations. File for traceability, then close as a duplicate of #1172 unless the episode never drains (which would point to a leaked slot and warrant its own investigation).
Evidence
| Fingerprint | 52e3fc831213ea4d |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex |
| Pods | memex-portal-deployment-5998d5fddf-9sfsv |
| Occurrences | 2 |
| First seen | 2026-09-21 10:01:24Z |
| Last seen | 2026-09-21 10:01:24Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-21 10:01:24Z memex-portal-deployment-5998d5fddf-9sfsv crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [c0d85529#1 started 2026-09-21T10:01:24.8564046Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 4, deepest per-destination queue 36, routing pool subscribing 0. Latest dispatch target portal/nodeops-9VkLPkBrRU-rwTJ4WyrfiQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 10:01:24Z memex-portal-deployment-5998d5fddf-9sfsv crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [c0d85529#1 started 2026-09-21T10:01:24.8564046Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 4, deepest per-destination queue 36, routing pool subscribing 0. Latest dispatch target portal/nodeops-9VkLPkBrRU-rwTJ4WyrfiQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/52e3fc831213ea4d. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 21, 2026
Closes as duplicate of #5009.
Reason: same defect —
RoutingGraindispatch slots saturate at the 64-in-flight reporting threshold with legs head-of-line blocked behind a single destination. Grouped on the discriminator the log line itself defines, not on the title: a deepest per-destination queue of 1 or more means legs are blocked behind a leg. This ticket measured deepest per-destination queue 36 of a 64-slot budget over 4 queued destination(s) — so the great majority of the budget was held behind one leg, which is #5009's subject and not the empty-queue load shape tracked separately at #5018.Evidence: namespace
memex, 2 occurrence(s), 2026-09-21 10:01:24Z → 2026-09-21 10:01:24Z, episode stamp(s)c0d85529#1, latest dispatch target(s)portal/nodeops-9VkLPkBrRU-rwTJ4WyrfiQ. Fingerprint52e3fc831213ea4d.Unique evidence — migrated to #5009: destination
portal/nodeops-9VkLPkBrRU-rwTJ4WyrfiQ, the same id as #5137/#5145'scache/form — a third client pair showing the paired-address pattern.