RoutingGrain: route dispatch slots leak on a stuck leg, head-of-line blocking one cache destination #5130

What is failing

The Orleans RoutingGrain (MeshWeaver.Hosting) crossed its back-pressure threshold with 64 route dispatches in flight. Critically, 62 of the 64 in-flight legs were queued behind a single stream destination (cache/wpWb1fSJPUacoVxY6LnltQ) — this is head-of-line blocking on one destination, not general routing load.

Probable cause (moderate confidence)

One route leg to that destination never terminated, so its slot leaked and every subsequent dispatch to the same destination queued behind it. The log's own guidance supports this reading: destinations queued 1, deepest per-destination queue 62 means legs are blocked behind a leg, and with only one higher episode or a drain line on activation d6013194 would the episode clear. Only a single occurrence is fingerprinted here, so we cannot see whether a clearing line followed — if no higher episode ever appeared on that activation, the in-flight count never fell below half the threshold and a leg slot leaked. Alternative cause worth ruling out: a CPU-starved silo holding slots in the unbounded ThreadPool wait before leg timeouts start (the log text explicitly calls this out as a false positive shape). Check raw logs for later lines from activation d6013194: a higher episode ⇒ drained; no follow-up ⇒ leaked slot.

Impact

Single occurrence, single pod, at 09:09 UTC on 2026-09-21. Delays to updates routed to the cache/wpWb1fSJPUacoVxY6LnltQ destination while the backlog held; no evidence yet of a persistent leak. However, this is a recurring family: at least a dozen sibling back-pressure incidents from the same log site appear in the incident index over the past days, so the pattern deserves a systemic look rather than one-off triage.

Where to look

MeshWeaver.Hosting.Orleans.RoutingGrain — the route dispatch/leg lifecycle: the slot-acquisition and release paths, and the per-destination ordered queue for cache/… destinations. Specifically verify that a slot is unconditionally released when a leg faults or times out, including during the pre-timeout ThreadPool wait. The long diagnostic text in the log line itself documents the intended semantics (episode stamps, half-threshold drain rule) — use it as the spec.

Possible duplicate: many sibling incidents share this fingerprint shape (e.g. Admin/_LogIncident/03e5e31ad393af55, 32a1cc19191c2e6b, 84f199c67175e339); a human may fold them into one investigation.

Evidence

2026-09-21 09:09:24Z memex-portal-deployment-5998d5fddf-75t7b crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [d6013194#1 started 2026-09-21T09:09:24.2783740Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/wpWb1fSJPUacoVxY6LnltQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/72a4aed3f6030520. Recurrences are folded into this issue rather than opening new ones.}