RoutingGrain dispatch pool starved of slots on memex-cloud portal: 52 legs waiting for a pool slot while dispatches stall >22s #5418

What is failing

The Orleans RoutingGrain on the memex-cloud portal pod hit its dispatch threshold (64 in flight) with 52 legs waiting for a routing-pool slot while only 1 was inside its subscribe prologue — i.e. the routing pool has run out of slots and dispatches are queueing before they can even begin their subscribe. The oldest leg had been in flight for ~22.6 s (dispatch → Publish), so route deliveries on this activation were stalled well beyond anything attributable to a young, absorbing burst.

Probable cause

Medium confidence: routing-pool slot exhaustion on this pod. Per the log site's own diagnostics (developed in core issues #5009 and #5018), a high waiting for a pool slot with a near-zero subscribing gauge is the signature of a slot/THREAD shortage — legs are parked waiting for a pool slot before their subscribe, and the slot hold includes the unbounded wait for a ThreadPool thread. A single line cannot fully discriminate load from a leaked slot; if no higher-episode line ever follows for activation 49e69485, the slot never came back and this is a leak rather than load. With only one occurrence recorded so far, that discriminator is not yet available.

Impact

Low so far: 1 occurrence on 1 pod (memex-portal-deployment-857546649-vf2q6) at 22:55 UTC on 2026-09-22, one episode. If the slots are leaking rather than merely saturated, subsequent episodes on this activation will not clear and route delivery latency on that pod degrades progressively — worth watching, not currently user-visible at scale.

Where to look

• src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs — slot acquisition and release around Publish dispatch legs; check whether a leg that fails or is cancelled before its subscribe prologue releases its slot.
• Prior art on this log site: core issues RoutingGrain dispatch slots saturate with legs head-of-line blocked behind one shared-cache destination while the second pod starves on load #5009 (head-of-line blocking, fixed in PR 5151), RoutingGrain dispatch pool saturates repeatedly on portal silos; slots held through unbounded ThreadPool wait and episode-stamp splitting floods incidents #5018 (pool gauge semantics), RoutingGrain back-pressure, thread-shortage shape (waiting-for-slot high, subscribing ≤1): compile-off-ThreadPool fix #5327/#5375 live on memex-cloud, pending on memex; post-fix residual under verification #5118 (alerting verdict), and feedback rbuergi/Feedback/route-backpressure-fingerprint-splits-20260921-2236.

Note — near-duplicate: this incident belongs to the same family as issue #5273 (filed from sibling incident Admin/_LogIncident/449283692ef59065), which covers the normaliser failing to mask the [activation#episode started timestamp] group so this log site splits into dozens of near-identical Critical incidents. A human can likely close this as a duplicate of #5273; it is filed so the distinct pool-slot-wait reading on this pod's line (52 waiting / 1 subscribing / 22.6 s oldest) is not lost in the fold.

Evidence

2026-09-22 22:55:21Z memex-portal-deployment-857546649-vf2q6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [49e69485#4 started 2026-09-22T22:55:21.3358796Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0, routing pool subscribing 1, waiting for a pool slot 52; oldest leg in flight 22574 ms — dispatch → Publish (delivery ZKI0VTPckEefsUF4-i3jOQ). Latest dispatch target Store/Install — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an …[truncated]

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/f8bff043da94e11d. Recurrences are folded into this issue rather than opening new ones.}