Skip to content

RoutingGrain dispatches to dead pod hub cache/PlOy4bo4wUGD_muJVcJzcg serialize into a 62-deep head-of-line queue #5000

Closed as duplicate

Description

@systemorph-com

What is failing

The mesh routing grain (MeshWeaver.Hosting.Orleans.RoutingGrain) repeatedly hits its 64-dispatch back-pressure threshold with a genuine head-of-line blockage: 62 dispatches queue behind a single leg targeting the pod hub destination cache/PlOy4bo4wUGD_muJVcJzcg, on a recurring ~37-minute cadence, identically each time (same destination, same depth of 62).

Probable cause

Moderate confidence: the pod hub at cache/PlOy4bo4wUGD_muJVcJzcg is intermittently unavailable or extremely slow (refused/dead grain), and because dispatches to a destination are serialized, one blocked leg stalls every other dispatch to that destination until its own timeout fires. The identical depth and destination each episode suggests a periodic dispatcher keeps re-triggering against a dead hub rather than random load. Supporting evidence: on 2026-09-14 the same namespace logged repeated [ROUTE] Directed delivery to pod hub 'cache/…' was refused entries (see Ops/Logs/memex-1789416031191996185-6b6b44-klrt7 and neighbours), so unreachable cache pod hubs have prior form. Note the sibling pod's crossings (deepest queue 0, episodes advancing) are ordinary busy-silo load by the alert's own semantics and are probably not part of the defect.

Impact

8 critical crossings on 2 pods over ~2 hours. Episodes do drain (episode numbers advance), so this is not a slot leak — but each episode stalls up to 62 dispatches to one cache destination for the duration of a leg timeout, delaying cache-bound updates in the memex namespace until the episode clears.

Where to look

  • The per-destination dispatch queue / leg serialization in RoutingGrain (category MeshWeaver.Hosting.Orleans.RoutingGrain) — especially how a refused or unresponsive destination leg holds the head-of-line slot.
  • The state of pod hub cache/PlOy4bo4wUGD_muJVcJzcg on pod memex-portal-deployment-5f69f6b6bd-74xjv — whether it exists, answers, and how delivery refusals are surfaced.
  • Background: Doc/WhatsNew/2026-08-12-routing-alerts-name-what-they-actually-saw — the alert semantics (episode stamps, deepest-queue meaning) this reading relies on.

Suggested first step: grep the routing logs around each crossing for a directed delivery … was refused or timeout line naming cache/PlOy4bo4wUGD_muJVcJzcg, and check whether legs to a refused destination back off instead of head-of-line blocking.


Evidence

Fingerprint 696ac84fcbdb9880
Category MeshWeaver.Hosting.Orleans.RoutingGrain
Severity Critical
Namespace memex
Pods memex-portal-deployment-5f69f6b6bd-z7tb6, memex-portal-deployment-5f69f6b6bd-74xjv
Occurrences 8
First seen 2026-09-20 17:10:59Z
Last seen 2026-09-20 19:09:59Z
Routing not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines
2026-09-20 17:10:59Z memex-portal-deployment-5f69f6b6bd-z7tb6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [a0e2984b#1 started 2026-09-20T17:10:59.2370130Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 1. Latest dispatch target Publish/Deck — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 17:10:59Z memex-portal-deployment-5f69f6b6bd-74xjv crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [871d4cfb#12 started 2026-09-20T17:10:59.2461610Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/PlOy4bo4wUGD_muJVcJzcg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 17:47:59Z memex-portal-deployment-5f69f6b6bd-z7tb6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [a0e2984b#2 started 2026-09-20T17:47:59.2409355Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 1. Latest dispatch target Signature/SkribbleCredential — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 17:47:59Z memex-portal-deployment-5f69f6b6bd-74xjv crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [871d4cfb#13 started 2026-09-20T17:47:59.2470999Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/PlOy4bo4wUGD_muJVcJzcg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 18:10:59Z memex-portal-deployment-5f69f6b6bd-z7tb6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [a0e2984b#3 started 2026-09-20T18:10:59.2329639Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Store/Licensing — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 18:10:59Z memex-portal-deployment-5f69f6b6bd-74xjv crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [871d4cfb#14 started 2026-09-20T18:10:59.2446089Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/PlOy4bo4wUGD_muJVcJzcg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 19:09:59Z memex-portal-deployment-5f69f6b6bd-z7tb6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [a0e2984b#4 started 2026-09-20T19:09:59.2383463Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target UWDeepfield/UWDeepfieldHome — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 19:09:59Z memex-portal-deployment-5f69f6b6bd-74xjv crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [871d4cfb#16 started 2026-09-20T19:09:59.2458931Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/PlOy4bo4wUGD_muJVcJzcg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

Opened automatically from Admin/_LogIncident/696ac84fcbdb9880. Recurrences are folded into this issue rather than opening new ones.

Activity

  1. rbuergi commented on Sep 21, 2026

    @rbuergi
    Contributor

    Closes as duplicate of #5009 — Migrated in full — this carried the sharpest hypothesis in the set: a ~37-minute cadence with identical destination and identical depth 62 every time, read as a periodic dispatcher re-triggering against a dead pod hub, plus the prior [ROUTE] Directed delivery to pod hub 'cache/…' was refused entries from 2026-09-14. Now §4 of the canonical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions