Repository navigation
RoutingGrain dispatch slots saturate on cache destination — possible leaked leg blocking route dispatch #5145
Description
What is failing
The Orleans RoutingGrain (MeshWeaver.Hosting.Orleans) hit its route-dispatch back-pressure threshold: 64 dispatches in flight on activation 67930f71, episode #1, with 3 stream destinations queued and a deepest per-destination queue of 24. One destination — cache/9VkLPkBrRU-rwTJ4WyrfiQ — is holding a head-of-line backlog while other legs pile up behind it.
Probable cause
Evidence points at a single slow or stuck leg for the cache/... destination: a per-destination queue depth of 24 means legs are blocked behind one leg, and routing pool subscribing 0 indicates no new subscriptions are draining the stream side either. However, confidence is low — the log site itself warns the depth is sampled at the threshold crossing and is partly an artefact. Two interpretations the log distinguishes and the data does not yet resolve:
- If a later line on activation
67930f71shows a higher episode, this episode drained and this is transient load. - If neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — i.e. a leg never terminated and its dispatch slot leaked, which would be a genuine routing-slot leak in
RoutingGrain.
Neither follow-up line is present in the incident window, so the slot-leak hypothesis cannot be ruled out. Note also that the slot is held from dispatch until leg termination including the unbounded wait for a ThreadPool thread, so a CPU-starved silo would raise this with nothing genuinely stuck — check silo thread-pool saturation for 10:17 UTC before assuming a code defect.
Impact
Minimal so far: 2 occurrences, a single timestamp (2026-09-21 10:17 UTC), one pod (memex-portal-deployment-5998d5fddf-9sfsv). No evidence of recurrence in the ~7.5 hours since. If the slot-leak interpretation is right, each recurrence permanently consumes dispatch capacity on the affected activation until the grain is recycled — worth confirming before dismissing.
Where to look
The log site is MeshWeaver.Hosting.Orleans.RoutingGrain (category), event id 0, [ROUTE] back-pressure episode reporting. Start from the leg-termination path that releases a dispatch slot, and the ThreadPool wait that precedes the leg's own timeouts. Search the memex pod's logs around 2026-09-21T10:17:24Z for any later 67930f71# episode lines to decide between transient load and a leaked slot. The RoutingGrain source was not found in the readable mesh, so no direct file link is available.
Evidence
| Fingerprint | 82e3f3497ab75e30 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex |
| Pods | memex-portal-deployment-5998d5fddf-9sfsv |
| Occurrences | 2 |
| First seen | 2026-09-21 10:17:24Z |
| Last seen | 2026-09-21 10:17:24Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-21 10:17:24Z memex-portal-deployment-5998d5fddf-9sfsv crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [67930f71#1 started 2026-09-21T10:17:24.2573162Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 3, deepest per-destination queue 24, routing pool subscribing 0. Latest dispatch target cache/9VkLPkBrRU-rwTJ4WyrfiQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 10:17:24Z memex-portal-deployment-5998d5fddf-9sfsv crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [67930f71#1 started 2026-09-21T10:17:24.2573162Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 3, deepest per-destination queue 24, routing pool subscribing 0. Latest dispatch target cache/9VkLPkBrRU-rwTJ4WyrfiQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/82e3f3497ab75e30. Recurrences are folded into this issue rather than opening new ones.
Activity
systemorph-com commentedon Sep 21, 2026
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-5998d5fddf-9sfsvRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 21, 2026
Closes as duplicate of #5009.
Reason: same defect —
RoutingGraindispatch slots saturate at the 64-in-flight reporting threshold with legs head-of-line blocked behind a single destination. Grouped on the discriminator the log line itself defines, not on the title: a deepest per-destination queue of 1 or more means legs are blocked behind a leg. This ticket measured deepest per-destination queue 24 of a 64-slot budget over 3 queued destination(s) — so the great majority of the budget was held behind one leg, which is #5009's subject and not the empty-queue load shape tracked separately at #5018.Evidence: namespace
memex, 2 occurrence(s), 2026-09-21 10:17:24Z → 2026-09-21 10:17:24Z, episode stamp(s)67930f71#1, latest dispatch target(s)cache/9VkLPkBrRU-rwTJ4WyrfiQ. Fingerprint82e3f3497ab75e30.