RoutingGrain: head-of-line blocking on a single cache destination drove dispatches to the 64-slot threshold #5029

What is failing

A single RoutingGrain activation on the memex portal pod hit the 64-dispatch back-pressure threshold in one crossing. The tell is the per-destination breakdown: 64 legs spread over only 3 stream destinations, with a deepest per-destination queue of 25 — meaning ~25 route legs were queued behind a single leg to the same destination (head-of-line blocking), the one variant of this alert the instrumented text explicitly identifies as a genuine blockage rather than mere load. The blocked destination was the cache/… stream the sample names.

Probable cause

One route leg to that destination stalled (most likely waiting on a ThreadPool thread or a slow subscriber), and per-destination ordering then queued every other dispatch to that destination behind it. Confidence: moderate — the depth of 25 at the crossing is strong evidence of head-of-line blocking, but the sample is taken at threshold-crossing and only one line exists in this burst, so we cannot yet distinguish "slow leg" from "slot leaked and never terminated" (the alert's episode-stamp guidance: if no higher-episode or cleared line follows this activation, a slot leaked and the leg never terminated). Worth checking whether a subsequent cleared line for activation b4b7be02 exists in the pod's log before triaging further.

Impact

Low-to-moderate: one occurrence on one pod over the incident window. No user-facing outage is evidenced; the affected stream (cache destination) would have seen delayed deliveries for the duration of the episode. If episodes recur or never drain, impact escalates to that destination being effectively stalled.

Where to look

• MeshWeaver.Hosting.Orleans.RoutingGrain — the dispatch/slot accounting and per-destination queue logic (a slot is held from dispatch until the leg terminates, including the unbounded ThreadPool wait before the leg's own timeouts).
• Prior work on this exact log site: Doc/WhatsNew/2026-08-12-routing-alerts-name-what-they-actually-saw — that fix already addressed shutdown-cancelled legs leaking slots; verify whether a leg cancelled mid-episode (not at shutdown) can still terminate without releasing its slot.
• Historical warn-level crossings from the same site appear in Ops/Logs (2026-08-12, 2026-09-10, 2026-09-14) — worth correlating whether the same destination recurs.

Evidence

2026-09-20 22:20:23Z memex-portal-deployment-6b7c4f94db-zzvfq crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [b4b7be02#1 started 2026-09-20T22:20:23.6614262Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 3, deepest per-destination queue 25, routing pool subscribing 0. Latest dispatch target cache/Xpi-G1Q-lU6s2Moll9rUcw — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/55a9ce7bfc40ee71. Recurrences are folded into this issue rather than opening new ones.}