Skip to content

Routing grain hits dispatch back-pressure threshold on both portal pods simultaneously; one pod shows a 62-deep head-of-line queue on a single cache destination #4900

Closed as duplicate
Closed as duplicate
Routing grain hits dispatch back-pressure threshold on both portal pods simultaneously; one pod shows a 62-deep head-of-line queue on a single cache destination#4900
Labels
area:orleansCore subsystem: the Orleans silo - grains, placement, routingbugSomething isn't workingfeature:routing-back-pressureFeature: routing back pressuresev:MMedium - secondary path broken, or easy workaround

Description

@systemorph-com
systemorph-com
bot
opened on Sep 19, 2026

What is failing

The Orleans routing grain on both memex-portal pods crossed the 64-dispatch back-pressure threshold within 11 ms of each other, so this is one platform-wide event, not two coincidences.

Probable cause — unclear, two distinct shapes

The two samples say different things, and neither alone names a defect:

  1. Load, not blockage. On one pod (episode Add entire company to Open SMC Teams #1 on a fresh activation, all per-destination queues 0), the in-flight count hit the threshold with nothing waiting on anything. Per the alert's own semantics that is a burst of dispatches outpacing leg completion — typically CPU starvation while legs wait unboundedly for a ThreadPool thread before their own timeouts start. No destination is at fault.
  2. Genuine head-of-line blockage. On the other pod (episode Update release-packages.yml #19), 62 of the 64 in-flight legs are queued behind a single leg to one cache/… destination. That is a real blockage signature: one leg to that destination is not terminating and everything else for it is stuck behind it.

Confidence: medium that shape 2 is a true head-of-line block on that cache destination; low on what ties the two pods together — a shared trigger at the same instant (a platform-wide fan-out) is plausible but unevidenced. Crucially, it cannot yet be told whether the blocked leg ever terminated: a follow-up ROUTE line with a higher episode on the same activation means the episode drained; no follow-up at all means a slot leaked and the leg never finished. Please pull the raw log around the timestamps for activations fcf31578 (and cfb5d696) before diagnosing further — that is the deciding evidence, and it is not in this incident.

Impact

One critical event spanning both portal pods at a single instant. Route dispatches may have been delayed for its duration; there is no evidence yet of a sustained outage or slot leak, but the head-of-line shape on the cache destination is the one worth confirming before it recurs under load.

Where to look

  • Slot/leg accounting and the per-destination queue in MeshWeaver.Hosting.Orleans.RoutingGrain — specifically why a leg to a cache/… hub can hold 62 legs behind it.
  • The leg-termination path for cache-hub destinations; note Ops logs from 2026-09-14 show cache-hub destinations refusing directed deliveries — check whether a cache hub that stops answering leaves its leg hanging (refused ≠ terminated).
  • Background: Routing back-pressure alerts now name what they actually saw — the slot-leak-at-shutdown fix and alert semantics this incident's reading depends on.

Evidence
Fingerprint 5b6a59648c91fd85
Category MeshWeaver.Hosting.Orleans.RoutingGrain
Severity Critical
Namespace memex-cloud
Pods memex-portal-deployment-69956b6dbc-s246c, memex-portal-deployment-69956b6dbc-gx6z6
Occurrences 2
First seen 2026-09-19 14:36:47Z
Last seen 2026-09-19 14:36:47Z
Recent log lines 
2026-09-19 14:36:47Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [cfb5d696#1 started 2026-09-19T14:36:47.4884463Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target SST/_Issue/201 — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 14:36:47Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#19 started 2026-09-19T14:36:47.4993840Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/YORFhwiyqEyTP9ESF-NUNg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

Opened automatically from Admin/_LogIncident/5b6a59648c91fd85. Recurrences are folded into this issue rather than opening new ones.

Activity

  1. systemorph-com
    added
    bugSomething isn't working
    on Sep 19, 2026

  2. systemorph-com commented on Sep 19, 2026

    @systemorph-com
    systemorph-comBot
    on Sep 19, 2026
    ContributorAuthor

    Still occurring: 2 more since the last update (4 total).

    • Last seen: 2026-09-19 14:57:47Z
    • Pods: memex-portal-deployment-69956b6dbc-s246c, memex-portal-deployment-69956b6dbc-gx6z6
    Recent log lines 
    2026-09-19 14:36:47Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#19 started 2026-09-19T14:36:47.4993840Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/YORFhwiyqEyTP9ESF-NUNg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 14:57:47Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [cfb5d696#3 started 2026-09-19T14:57:47.4566566Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target ILS/Tranche — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 14:57:47Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#26 started 2026-09-19T14:57:47.4763151Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/YORFhwiyqEyTP9ESF-NUNg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

    Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  3. rbuergi commented on Sep 19, 2026

    @rbuergi
    rbuergi
    on Sep 19, 2026
    Contributor

    Closing as a duplicate of #4798.

    This issue and 46 others were filed today by one log site — MeshWeaver.Hosting.Orleans.RoutingGrain's back-pressure line — because the Orleans activation id in that line was not masked before the fingerprint was taken. LogLineParser.HexBlob required sixteen hex characters; an activation id is eight, so it fell through to the number masker, which stripped its digits and left its letters behind as literal text. An activation id changes on every grain recycle and every pod roll, and memex-cloud rolled repeatedly today, so each activation minted a new fingerprint, a new incident and a new issue.

    These are not 47 defects. Under correct masking all 47 normalize to one message and would have been one incident. Verified two of them against the control instance's own records: Admin/_LogIncident/5b6a59648c91fd85 and eb29857024b3f0ed differ only in cfb5d6 vs c0cc8 — the dispatch target is already {path} in both, and every count is already {n}.

    Fixed in Systemorph/MeshWeaver.Plugins#2165 (merged), which masks a run of ≥ 8 hex characters carrying both a digit and an a–f letter. The fault itself — routing back-pressure at the 64-dispatch threshold — remains open and is tracked on #4798, which carries the longest occurrence history.

    Reopen if you believe this one is a distinct fault rather than another activation of the same line.

  4. rbuergi
    closed this as not plannedon Sep 19, 2026

  5. systemorph-com commented on Sep 19, 2026

    @systemorph-com
    systemorph-comBot
    on Sep 19, 2026
    ContributorAuthor

    Reopened — this fault is happening again.

    Still occurring: 5 more since the last update (9 total).

    • Last seen: 2026-09-19 15:51:47Z
    • Pods: memex-portal-deployment-69956b6dbc-s246c, memex-portal-deployment-69956b6dbc-gx6z6, memex-portal-deployment-69956b6dbc-ndnxt
    Recent log lines 
    2026-09-19 15:27:47Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [c0cc8549#19 started 2026-09-19T15:27:47.4747810Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/YORFhwiyqEyTP9ESF-NUNg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 15:51:47Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [cfb5d696#8 started 2026-09-19T15:51:47.4889980Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 2. Latest dispatch target RemoteControl/Screen — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 15:51:47Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [fcf31578#37 started 2026-09-19T15:51:47.4928054Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/YORFhwiyqEyTP9ESF-NUNg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

    Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  6. systemorph-com
    reopened this on Sep 19, 2026
  7. rbuergi
    added
    area:orleansCore subsystem: the Orleans silo - grains, placement, routing
    feature:routing-back-pressureFeature: routing back pressure
    sev:MMedium - secondary path broken, or easy workaround
    on Sep 20, 2026

  8. rbuergi commented on Sep 21, 2026

    @rbuergi
    rbuergi
    on Sep 21, 2026
    Contributor

    Closes as duplicate of #5009 — Migrated: both portal pods crossed the threshold within 11 ms of each other — one platform-wide event, not two coincidences — plus the refused ≠ terminated reading about cache-hub delivery refusals. Both now in §6 and §4 of the canonical.

  9. rbuergi
    closed this as duplicateon Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:orleansCore subsystem: the Orleans silo - grains, placement, routingbugSomething isn't workingfeature:routing-back-pressureFeature: routing back pressuresev:MMedium - secondary path broken, or easy workaround

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions