RoutingGrain head-of-line blocking: route dispatches pile up behind a single stuck destination leg #5092

What is failing

The Orleans routing grain (MeshWeaver.Hosting.Orleans.RoutingGrain) hit its back-pressure threshold (64 in-flight route dispatches) with a deepest per-destination queue of 62 out of 64 in flight — i.e. essentially the entire backlog is stacked behind a single destination (dispatch target cache/9mdycNhVAE6S3qduJCUHVA). This is the classic head-of-line signature the log site itself describes: one leg to one destination is blocked, and every other dispatch to that destination is queued behind it.

Probable cause

One routing leg to the cache/... destination did not terminate, and ~62 subsequent dispatches to the same destination piled up behind it. The log's own guidance says the in-flight count at crossing is partly a threshold artefact, but with destinations queued = 1 and deepest queue = 62, this is not diffuse load — it is a single blocked destination.

Two candidate mechanisms, in order of likelihood:

1. Head-of-line blocking on a slow/dead destination — the leg is stuck waiting (e.g. on a ThreadPool thread before its own timeouts start, per the log text), and the per-destination queue grows behind it.
2. Slot leak — if the episode never drains (no later line with a higher episode on the same activation and no different activation id), a leg's slot is held forever because it never terminated.

Confidence: low-to-moderate. There is a single occurrence with no exception or stack, and no follow-up episode line was captured to distinguish "drained slowly" from "leaked permanently" — that follow-up evidence is the first thing an engineer should pull from the pod's log around 2026-09-20T23:12:58Z (activation fee04716, episode #2).

Impact

Single occurrence, single pod (memex-portal-deployment-75b545bbd-gwrc6), ~15h before filing. At the moment of the event, route dispatches to that destination were delayed behind the blocked leg; there is no evidence yet of an outage or persistent degradation. Treat as an investigation ticket, not an emergency — unless the episode never drained, in which case a leaked slot makes the routing grain permanently slower on that activation.

Where to look

• MeshWeaver.Hosting.Orleans.RoutingGrain — the dispatch/leg lifecycle: where a slot is acquired at dispatch and released at leg termination, and the per-destination queue feeding legs.
• The destination cache/9mdycNhVAE6S3qduJCUHVA — check what consumer sits there and whether it was slow or wedged around 23:12 UTC on 2026-09-20.
• First diagnostic step: search the pod's logs for the same activation id fee04716 — a higher episode number means it drained; a different activation id means the grain was recycled; neither means a slot leaked.

Evidence

2026-09-20 23:12:58Z memex-portal-deployment-75b545bbd-gwrc6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fee04716#2 started 2026-09-20T23:12:58.3788139Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/9mdycNhVAE6S3qduJCUHVA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/12c8da80f8989bd0. Recurrences are folded into this issue rather than opening new ones.}