Repository navigation
RoutingGrain dispatches to dead pod hub cache/PlOy4bo4wUGD_muJVcJzcg serialize into a 62-deep head-of-line queue #5000
Description
What is failing
The mesh routing grain (MeshWeaver.Hosting.Orleans.RoutingGrain) repeatedly hits its 64-dispatch back-pressure threshold with a genuine head-of-line blockage: 62 dispatches queue behind a single leg targeting the pod hub destination cache/PlOy4bo4wUGD_muJVcJzcg, on a recurring ~37-minute cadence, identically each time (same destination, same depth of 62).
Probable cause
Moderate confidence: the pod hub at cache/PlOy4bo4wUGD_muJVcJzcg is intermittently unavailable or extremely slow (refused/dead grain), and because dispatches to a destination are serialized, one blocked leg stalls every other dispatch to that destination until its own timeout fires. The identical depth and destination each episode suggests a periodic dispatcher keeps re-triggering against a dead hub rather than random load. Supporting evidence: on 2026-09-14 the same namespace logged repeated [ROUTE] Directed delivery to pod hub 'cache/…' was refused entries (see Ops/Logs/memex-1789416031191996185-6b6b44-klrt7 and neighbours), so unreachable cache pod hubs have prior form. Note the sibling pod's crossings (deepest queue 0, episodes advancing) are ordinary busy-silo load by the alert's own semantics and are probably not part of the defect.
Impact
8 critical crossings on 2 pods over ~2 hours. Episodes do drain (episode numbers advance), so this is not a slot leak — but each episode stalls up to 62 dispatches to one cache destination for the duration of a leg timeout, delaying cache-bound updates in the memex namespace until the episode clears.
Where to look
- The per-destination dispatch queue / leg serialization in
RoutingGrain(categoryMeshWeaver.Hosting.Orleans.RoutingGrain) — especially how a refused or unresponsive destination leg holds the head-of-line slot. - The state of pod hub
cache/PlOy4bo4wUGD_muJVcJzcgon podmemex-portal-deployment-5f69f6b6bd-74xjv— whether it exists, answers, and how delivery refusals are surfaced. - Background: Doc/WhatsNew/2026-08-12-routing-alerts-name-what-they-actually-saw — the alert semantics (episode stamps, deepest-queue meaning) this reading relies on.
Suggested first step: grep the routing logs around each crossing for a directed delivery … was refused or timeout line naming cache/PlOy4bo4wUGD_muJVcJzcg, and check whether legs to a refused destination back off instead of head-of-line blocking.
Evidence
| Fingerprint | 696ac84fcbdb9880 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex |
| Pods | memex-portal-deployment-5f69f6b6bd-z7tb6, memex-portal-deployment-5f69f6b6bd-74xjv |
| Occurrences | 8 |
| First seen | 2026-09-20 17:10:59Z |
| Last seen | 2026-09-20 19:09:59Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-20 17:10:59Z memex-portal-deployment-5f69f6b6bd-z7tb6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [a0e2984b#1 started 2026-09-20T17:10:59.2370130Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 1. Latest dispatch target Publish/Deck — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 17:10:59Z memex-portal-deployment-5f69f6b6bd-74xjv crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [871d4cfb#12 started 2026-09-20T17:10:59.2461610Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/PlOy4bo4wUGD_muJVcJzcg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 17:47:59Z memex-portal-deployment-5f69f6b6bd-z7tb6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [a0e2984b#2 started 2026-09-20T17:47:59.2409355Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 1. Latest dispatch target Signature/SkribbleCredential — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 17:47:59Z memex-portal-deployment-5f69f6b6bd-74xjv crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [871d4cfb#13 started 2026-09-20T17:47:59.2470999Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/PlOy4bo4wUGD_muJVcJzcg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 18:10:59Z memex-portal-deployment-5f69f6b6bd-z7tb6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [a0e2984b#3 started 2026-09-20T18:10:59.2329639Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Store/Licensing — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 18:10:59Z memex-portal-deployment-5f69f6b6bd-74xjv crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [871d4cfb#14 started 2026-09-20T18:10:59.2446089Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/PlOy4bo4wUGD_muJVcJzcg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 19:09:59Z memex-portal-deployment-5f69f6b6bd-z7tb6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [a0e2984b#4 started 2026-09-20T19:09:59.2383463Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target UWDeepfield/UWDeepfieldHome — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 19:09:59Z memex-portal-deployment-5f69f6b6bd-74xjv crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [871d4cfb#16 started 2026-09-20T19:09:59.2458931Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/PlOy4bo4wUGD_muJVcJzcg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/696ac84fcbdb9880. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 21, 2026
Closes as duplicate of #5009 — Migrated in full — this carried the sharpest hypothesis in the set: a ~37-minute cadence with identical destination and identical depth 62 every time, read as a periodic dispatcher re-triggering against a dead pod hub, plus the prior
[ROUTE] Directed delivery to pod hub 'cache/…' was refusedentries from 2026-09-14. Now §4 of the canonical.