RoutingGrain back-pressure threshold crossed by transient dispatch burst — recurring site capacity should be tracked once, not per activation #5419

What is failing

The Orleans RoutingGrain (MeshWeaver.Hosting.Orleans) on the portal pod crossed its back-pressure reporting threshold: 64 route dispatches in flight, with 24 legs waiting for a routing-pool slot. The dispatch machinery itself did not fault — this is a capacity/back-pressure signal, not a crash.

Probable cause — transient burst load, absorbed (high confidence)

Every discriminator on the line points to load rather than a leak or a defect:

• Oldest leg in flight 263 ms — per the log site's own load-vs-leak rule, a young oldest leg means the silo was absorbing the burst; no slot appears stuck.
• Deepest per-channel queue 0 with 35 queued over 5 destinations — multiplexer-hub shape draining in parallel; no head-of-line blocking within any channel.
• routing pool subscribing 0 with waiting for a pool slot 24 — legs are queued before their subscribe prologue, i.e. waiting on ThreadPool capacity, not on downstream I/O.

Most likely a short burst of stream-routed dispatches (targets under cache/...) briefly exceeded the routing pool / ThreadPool headroom, crossed the 64-dispatch threshold once, and drained. Low confidence remains on whether the underlying thread-pool pressure is itself abnormal — see impact below.

Impact

Single occurrence on one pod (memex-portal-deployment-857546649-vf2q6), no delivery failure logged against it. On its own this is a nuisance. However, this log site has produced ~40 sibling back-pressure incidents between 2026-09-19 and 2026-09-23 (per-activation fingerprints, each opening its own incident). The systemic question an engineer should answer is whether the routing pool / silo ThreadPool sizing is adequate for the steady dispatch rate, and whether per-activation fingerprints for this site should be folded so one capacity ticket, not dozens, tracks it. If the aggregate rate is stable and episodes drain (higher episode stamps following), this is healthy load reporting.

Where to look

• MeshWeaver.Hosting.Orleans.RoutingGrain — the [ROUTE] Routing back-pressure log site and the routing-pool slot acquisition code (the pre-subscribe wait the waiting for a pool slot gauge measures).
• Prior related work referenced by the log itself: issues RoutingGrain dispatch slots saturate with legs head-of-line blocked behind one shared-cache destination while the second pod starves on load #5009 (per-channel fan-out) and RoutingGrain dispatch pool saturates repeatedly on portal silos; slots held through unbounded ThreadPool wait and episode-stamp splitting floods incidents #5018 (pool gauge semantics).
• Nearby symptoms in the same namespace that may share the thread-pressure root: OutOfMemoryException on route dispatch (incidents 0feb3c48c04cc4f1, 9539b56b6384fadb, 2026-09-19) and a burst of delivery-failure incidents on 2026-09-21/22.

Note on duplicates

This individual burst is almost certainly a duplicate symptom of the recurring back-pressure pattern tracked in the sibling incidents above; a human may close this in favour of a single capacity-tracking issue for the log site.

Evidence

2026-09-22 23:01:51Z memex-portal-deployment-857546649-vf2q6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [62c1244f#2 started 2026-09-22T23:01:51.4879698Z]: 64 route dispatches in flight (reporting threshold 64); ordered channels queued 35 over 5 stream destination(s), deepest per-channel queue 0, routing pool subscribing 0, waiting for a pool slot 24; oldest leg in flight 263 ms — stream-routed → cache/2v3dSLXHq0ekZ_MHIaE63g (delivery h0fj4nGWzEWVY8vW5Tm1Tg). Latest dispatch target cache/QQXFd9sRTUaPrj7FF36Gjg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A CHANNEL is (destination, stream), so a deepest queue of 1 or more means legs of the SAME stream are blocked behind one another (head-of-line within one channel); 0 means nothing is waiting on anything, so read it as load. Many channels over FEW destinations is a multiplexer hub draining in parallel, which is load too — before issue #5009 those legs shared one channel and read as head-of-line. 🚨 THE TWO POOL GAUGES ANSWER DIFFERENT QUESTIONS AND ONLY ONE SEES THE WAIT NAMED ABOVE — issue #5018. 'subscribing' counts legs inside their SUBSCRIBE PROLOGUE only, so a leg still waiting for a ThreadPool thread and a leg already past its subscribe BOTH read 0 there; 'waiting for a pool slot' is the gauge that sees the pre-subscribe wait. High while subscribing is below the pool's cap is a THREAD shortage; both near zero means the legs are past their subscribe and the wait is downstream I/O. 🚨 READ THE OLDEST LEG FIRST: it is the only load-vs-leak discriminator available from a SINGLE line — every leg young is load the silo is absorbing, one leg minutes old is a slot that never came back and the label names it — whereas the episode stamp below needs a SECOND line to say anything. 🚨 Deepest is sampled AT THE CROSSING, so l…[truncated]

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/7ca733d22f73ebd3. Recurrences are folded into this issue rather than opening new ones.}