Repository navigation
RoutingGrain dispatch slots back up behind a single slow stream destination (head-of-line blocking at the 64-in-flight threshold) #5133
Description
What is failing
The Orleans RoutingGrain in the memex portal hit its in-flight dispatch ceiling (64) with 61 of those dispatch legs queued behind a single stream destination (cache/wpWb1fSJPUacoVxY6LnltQ) — head-of-line blocking on one destination, not general routing load. Stream destinations queued were only 2 and the routing pool had 0 subscribers, so the backlog is concentrated on one slow/faulted leg while other traffic continues to consume slots and wait behind it.
Probable cause
One destination (cache/wpWb1fSJPUacoVxY6LnltQ) is not draining its stream legs fast enough (or not terminating them at all), so every dispatch to it piles up and holds its in-flight slot. Because a slot is held from dispatch until the leg terminates — including the unbounded wait for a ThreadPool thread before the leg's own timeouts start — this could equally be a CPU-starved silo backing up one destination rather than a fault in the destination itself. Moderate confidence: the depth concentration (61/64) clearly indicates single-destination head-of-line blocking, but with only one crossing and no follow-up episode/clear line yet captured, we cannot distinguish "slow destination" from "slot leak on a never-terminating leg". Checking whether activation 63637bff later logs a higher episode number (drained) or the in-flight count never falls below 32 (leaked leg) would settle it.
Impact
Low-to-moderate for this episode: one occurrence on a single pod (memex-portal-deployment-5998d5fddf-75t7b), a one-off crossing today. However, this is one of many back-pressure incidents filed from the same RoutingGrain log site in recent days (e.g. incidents 02dad774e67b99e5, 03e5e31ad393af55, 0669719574a72f1a, 0fc7665eaf6aae43, and others), so the site as a whole warrants a systemic look rather than a per-episode fix.
Where to look
MeshWeaver.Hosting.Orleans.RoutingGrain— the dispatch slot lifecycle: fromRoute dispatchto leg termination, especially the unbounded ThreadPool wait before leg timeouts begin. A timeout on that pre-dispatch wait would turn slow destinations into slot leaks.- The destination
cache/wpWb1fSJPUacoVxY6LnltQon the memex portal silo — why its legs terminate slowly (or not at all). - Per-destination queue drain logic: consider fairness or a per-destination cap so one slow destination cannot consume the whole 64-slot budget.
Evidence
| Fingerprint | 3c80fa8635aec16c |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex |
| Pods | memex-portal-deployment-5998d5fddf-75t7b |
| Occurrences | 2 |
| First seen | 2026-09-21 09:42:24Z |
| Last seen | 2026-09-21 09:42:24Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-21 09:42:24Z memex-portal-deployment-5998d5fddf-75t7b crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [63637bff#1 started 2026-09-21T09:42:24.8016348Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 2, deepest per-destination queue 61, routing pool subscribing 0. Latest dispatch target cache/wpWb1fSJPUacoVxY6LnltQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 09:42:24Z memex-portal-deployment-5998d5fddf-75t7b crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [63637bff#1 started 2026-09-21T09:42:24.8016348Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 2, deepest per-destination queue 61, routing pool subscribing 0. Latest dispatch target cache/wpWb1fSJPUacoVxY6LnltQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/3c80fa8635aec16c. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 21, 2026
Closes as duplicate of #5009.
Reason: same defect —
RoutingGraindispatch slots saturate at the 64-in-flight reporting threshold with legs head-of-line blocked behind a single destination. Grouped on the discriminator the log line itself defines, not on the title: a deepest per-destination queue of 1 or more means legs are blocked behind a leg. This ticket measured deepest per-destination queue 61 of a 64-slot budget over 2 queued destination(s) — so the great majority of the budget was held behind one leg, which is #5009's subject and not the empty-queue load shape tracked separately at #5018.Evidence: namespace
memex, 2 occurrence(s), 2026-09-21 09:42:24Z → 2026-09-21 09:42:24Z, episode stamp(s)63637bff#1, latest dispatch target(s)cache/wpWb1fSJPUacoVxY6LnltQ. Fingerprint3c80fa8635aec16c.Unique evidence — migrated to #5009: same blocked destination
cache/wpWb1fSJPUacoVxY6LnltQ; 61 of 64 slots behind one leg — the deepest single-destination reading in the burst.