Skip to content

[Duplicate of #5009] RoutingGrain dispatch budget saturated by head-of-line blocking on cache-destination legs across the portal fleet #5176

Closed as duplicate
Closed as duplicate
[Duplicate of #5009] RoutingGrain dispatch budget saturated by head-of-line blocking on cache-destination legs across the portal fleet#5176
Labels
bugSomething isn't workingduplicateThis issue or pull request already existssev:HBLOCKING: primary path broken, intermittent user-visible failure, or silently wrong results.

Description

@systemorph-com
systemorph-com
bot
opened on Sep 22, 2026

What is failing

The mesh router (MeshWeaver.Hosting.Orleans.RoutingGrain) on the memex-cloud portal fleet repeatedly saturates its dispatch budget: all 64 route slots go in flight, and in nearly every sampled episode 62 of the 64 legs are queued behind a single destination (stream destinations queued 1, deepest per-destination queue 62). That is head-of-line blocking on one destination, not load — one destination's leg holds the queue while ~62 other dispatches wait behind it.

Probable cause

Moderately confident: the blocked destinations in every readable sample are cache/<id> addresses (cache invalidation/fan-out subscribers). A single slow or wedged cache subscriber leg holds its destination queue, and since routing slots are held from dispatch until the leg terminates — including the unbounded wait for a ThreadPool thread — a slow cache leg (or thread-pool starvation triggered by it) pins the router. The episode stamps matter: samples show fresh episode ids roughly every 2–10 minutes with episode number #1, i.e. episodes drain and immediately recur rather than one permanently leaked slot (a leaked slot would show the same activation+episode with no higher episode ever following). So the router is repeatedly degraded, not permanently wedged — pointing at a recurring slow cache leg rather than the shutdown-cancellation slot leak fixed on 2026-08-12. Lower confidence on why the cache leg is slow (subscriber I/O vs ThreadPool starvation); the evidence does not distinguish those.

Impact

88 critical bursts over ~9.5 hours (2026-09-21 12:32–22:10 UTC) across all 8 portal pods. Each episode means the pod's routing capacity is fully consumed and non-cache destinations' dispatches queue behind the blocked leg — portal message delivery (including non-cache traffic) is intermittently delayed fleet-wide for the whole period. No evidence of data loss or permanent wedge; episodes clear.

Where to look

  • MeshWeaver.Hosting.Orleans.RoutingGrain — the per-destination leg queue and the slot-release path on leg termination; log site [ROUTE].
  • The cache destination subscriber (cache/* stream, e.g. cache/FtxSQnNgjkmJzr-09vwCJQ, cache/adQjx8eg8UyB5mWb_cRd5Q — both recur across pods) — what its legs wait on.
  • Prior art: Routing back-pressure alerts now name what they actually saw (slot-leak fix and alert semantics) and ActionBlockWedgePrevention.

Note: this is one of ~20 sibling back-pressure incidents opened on 2026-09-19..21 (e.g. Admin/_LogIncident/0fc7665eaf6aae43, Admin/_LogIncident/155d7ffcac9123d3) — likely the same underlying episode; consolidating them under this ticket is reasonable.

Evidence
Fingerprint 96c85f04cadba7f7
Category MeshWeaver.Hosting.Orleans.RoutingGrain
Severity Critical
Namespace memex-cloud
Pods memex-portal-deployment-77494bdbcc-7ndrn, memex-portal-deployment-7fc5f4874b-tl6zn, memex-portal-deployment-7fc5f4874b-9zwrg, memex-portal-deployment-77494bdbcc-bphfb, memex-portal-deployment-8d98c8fc6-w6clj, memex-portal-deployment-8d98c8fc6-f674p, memex-portal-deployment-7b4776b9f-mqfr6, memex-portal-deployment-7b4776b9f-8khld
Occurrences 88
First seen 2026-09-21 12:32:25Z
Last seen 2026-09-21 22:10:25Z
Routing not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-09-21 20:57:24Z memex-portal-deployment-8d98c8fc6-f674p fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
2026-09-21 20:59:24Z memex-portal-deployment-8d98c8fc6-f674p crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [cde6effa#1 started 2026-09-21T20:59:24.2992913Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 3, deepest per-destination queue 26, routing pool subscribing 1. Latest dispatch target cache/adQjx8eg8UyB5mWb_cRd5Q — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 21:07:24Z memex-portal-deployment-7b4776b9f-mqfr6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [cc8c953f#1 started 2026-09-21T21:07:24.3065069Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/rsFtZ2Qxwkmoo8tSu3Yzww — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 21:09:25Z memex-portal-deployment-77494bdbcc-7ndrn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [67cc298f#1 started 2026-09-21T21:09:25.1695035Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/mr9jbhbuyEGL3xTrvoGS9g — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 21:14:24Z memex-portal-deployment-8d98c8fc6-f674p crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [e02bc619#1 started 2026-09-21T21:14:24.3475160Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 3, deepest per-destination queue 26, routing pool subscribing 0. Latest dispatch target cache/FtxSQnNgjkmJzr-09vwCJQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 21:47:24Z memex-portal-deployment-7b4776b9f-8khld crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [707c966f#1 started 2026-09-21T21:47:24.3005854Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 1. Latest dispatch target cache/FtxSQnNgjkmJzr-09vwCJQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 21:59:24Z memex-portal-deployment-7b4776b9f-mqfr6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [000c904f#1 started 2026-09-21T21:59:24.2321318Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/adQjx8eg8UyB5mWb_cRd5Q — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 21:59:24Z memex-portal-deployment-7b4776b9f-mqfr6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [000c904f#1 started 2026-09-21T21:59:24.2321318Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/adQjx8eg8UyB5mWb_cRd5Q — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 22:10:24Z memex-portal-deployment-7b4776b9f-mqfr6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [aa98ed14#1 started 2026-09-21T22:10:24.2838546Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/adQjx8eg8UyB5mWb_cRd5Q — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 22:10:25Z memex-portal-deployment-77494bdbcc-7ndrn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [485b1c95#1 started 2026-09-21T22:10:25.2081129Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/xFMO23Y5r0uAuw3jmEtZpQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

Opened automatically from Admin/_LogIncident/96c85f04cadba7f7. Recurrences are folded into this issue rather than opening new ones.

Activity

  1. systemorph-com
    added
    bugSomething isn't working
    sev:HBLOCKING: primary path broken, intermittent user-visible failure, or silently wrong results.
    on Sep 22, 2026

  2. systemorph-com commented on Sep 22, 2026

    @systemorph-com
    systemorph-combot
    on Sep 22, 2026
    ContributorAuthor

    Still occurring: 1 more since the last update (89 total).

    • Last seen: 2026-09-21 22:23:24Z
    • Pods: memex-portal-deployment-77494bdbcc-7ndrn, memex-portal-deployment-7fc5f4874b-tl6zn, memex-portal-deployment-7fc5f4874b-9zwrg, memex-portal-deployment-77494bdbcc-bphfb, memex-portal-deployment-8d98c8fc6-w6clj, memex-portal-deployment-8d98c8fc6-f674p, memex-portal-deployment-7b4776b9f-mqfr6, memex-portal-deployment-7b4776b9f-8khld
    Recent log lines 
    2026-09-21 22:10:24Z memex-portal-deployment-7b4776b9f-mqfr6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [aa98ed14#1 started 2026-09-21T22:10:24.2838546Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/adQjx8eg8UyB5mWb_cRd5Q — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 22:10:25Z memex-portal-deployment-77494bdbcc-7ndrn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [485b1c95#1 started 2026-09-21T22:10:25.2081129Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/xFMO23Y5r0uAuw3jmEtZpQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 22:23:24Z memex-portal-deployment-77494bdbcc-bphfb crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [a44888c1#16 started 2026-09-21T22:23:24.8590449Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/xFMO23Y5r0uAuw3jmEtZpQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

    Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  3. systemorph-com commented on Sep 22, 2026

    @systemorph-com
    systemorph-combot
    on Sep 22, 2026
    ContributorAuthor

    Still occurring: 1 more since the last update (90 total).

    • Last seen: 2026-09-21 22:26:25Z
    • Pods: memex-portal-deployment-77494bdbcc-7ndrn, memex-portal-deployment-7fc5f4874b-tl6zn, memex-portal-deployment-7fc5f4874b-9zwrg, memex-portal-deployment-77494bdbcc-bphfb, memex-portal-deployment-8d98c8fc6-w6clj, memex-portal-deployment-8d98c8fc6-f674p, memex-portal-deployment-7b4776b9f-mqfr6, memex-portal-deployment-7b4776b9f-8khld
    Recent log lines 
    2026-09-21 22:10:25Z memex-portal-deployment-77494bdbcc-7ndrn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [485b1c95#1 started 2026-09-21T22:10:25.2081129Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/xFMO23Y5r0uAuw3jmEtZpQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 22:23:24Z memex-portal-deployment-77494bdbcc-bphfb crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [a44888c1#16 started 2026-09-21T22:23:24.8590449Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/xFMO23Y5r0uAuw3jmEtZpQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 22:26:25Z memex-portal-deployment-77494bdbcc-7ndrn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [1a3ac14f#1 started 2026-09-21T22:26:25.0498665Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/nuc_714_ykGlrZEjO5Vg9g — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

    Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  4. systemorph-com commented on Sep 22, 2026

    @systemorph-com
    systemorph-combot
    on Sep 22, 2026
    ContributorAuthor

    Still occurring: 1 more since the last update (91 total).

    • Last seen: 2026-09-21 22:30:24Z
    • Pods: memex-portal-deployment-77494bdbcc-7ndrn, memex-portal-deployment-7fc5f4874b-tl6zn, memex-portal-deployment-7fc5f4874b-9zwrg, memex-portal-deployment-77494bdbcc-bphfb, memex-portal-deployment-8d98c8fc6-w6clj, memex-portal-deployment-8d98c8fc6-f674p, memex-portal-deployment-7b4776b9f-mqfr6, memex-portal-deployment-7b4776b9f-8khld
    Recent log lines 
    2026-09-21 22:23:24Z memex-portal-deployment-77494bdbcc-bphfb crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [a44888c1#16 started 2026-09-21T22:23:24.8590449Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/xFMO23Y5r0uAuw3jmEtZpQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 22:26:25Z memex-portal-deployment-77494bdbcc-7ndrn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [1a3ac14f#1 started 2026-09-21T22:26:25.0498665Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/nuc_714_ykGlrZEjO5Vg9g — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 22:30:24Z memex-portal-deployment-77494bdbcc-7ndrn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [c02e6831#1 started 2026-09-21T22:30:24.5159023Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/nuc_714_ykGlrZEjO5Vg9g — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

    Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  5. systemorph-com commented on Sep 22, 2026

    @systemorph-com
    systemorph-combot
    on Sep 22, 2026
    ContributorAuthor

    Still occurring: 2 more since the last update (93 total).

    • Last seen: 2026-09-21 23:03:24Z
    • Pods: memex-portal-deployment-77494bdbcc-7ndrn, memex-portal-deployment-7fc5f4874b-tl6zn, memex-portal-deployment-7fc5f4874b-9zwrg, memex-portal-deployment-77494bdbcc-bphfb, memex-portal-deployment-8d98c8fc6-w6clj, memex-portal-deployment-8d98c8fc6-f674p, memex-portal-deployment-7b4776b9f-mqfr6, memex-portal-deployment-7b4776b9f-8khld
    Recent log lines 
    2026-09-21 22:30:24Z memex-portal-deployment-77494bdbcc-7ndrn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [c02e6831#1 started 2026-09-21T22:30:24.5159023Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/nuc_714_ykGlrZEjO5Vg9g — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 23:00:25Z memex-portal-deployment-77494bdbcc-7ndrn crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [1213baac#2 started 2026-09-21T23:00:25.3237386Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 2, deepest per-destination queue 48, routing pool subscribing 0. Latest dispatch target cache/xFMO23Y5r0uAuw3jmEtZpQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 23:03:24Z memex-portal-deployment-7b4776b9f-mqfr6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [7338f818#1 started 2026-09-21T23:03:24.3986962Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/adQjx8eg8UyB5mWb_cRd5Q — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

    Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  6. systemorph-com commented on Sep 22, 2026

    @systemorph-com
    systemorph-combot
    on Sep 22, 2026
    ContributorAuthor

    Still occurring: 3 more since the last update (96 total).

    • Last seen: 2026-09-21 23:06:24Z
    • Pods: memex-portal-deployment-77494bdbcc-7ndrn, memex-portal-deployment-7fc5f4874b-tl6zn, memex-portal-deployment-7fc5f4874b-9zwrg, memex-portal-deployment-77494bdbcc-bphfb, memex-portal-deployment-8d98c8fc6-w6clj, memex-portal-deployment-8d98c8fc6-f674p, memex-portal-deployment-7b4776b9f-mqfr6, memex-portal-deployment-7b4776b9f-8khld
    Recent log lines 
    2026-09-21 23:03:24Z memex-portal-deployment-7b4776b9f-mqfr6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [7338f818#1 started 2026-09-21T23:03:24.3986962Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/adQjx8eg8UyB5mWb_cRd5Q — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 23:05:24Z memex-portal-deployment-7b4776b9f-8khld crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [d361ad6e#1 started 2026-09-21T23:05:24.5387958Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/FtxSQnNgjkmJzr-09vwCJQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-21 23:06:24Z memex-portal-deployment-7b4776b9f-8khld crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [cb417a42#1 started 2026-09-21T23:06:24.5224356Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/FtxSQnNgjkmJzr-09vwCJQ — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

    Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  7. rbuergi
    changed the title [-]RoutingGrain dispatch budget saturated by head-of-line blocking on cache-destination legs across the portal fleet[/-] [+][Duplicate of #5009] RoutingGrain dispatch budget saturated by head-of-line blocking on cache-destination legs across the portal fleet[/+] on Sep 22, 2026
  8. rbuergi
    closed this as not plannedon Sep 22, 2026
  9. rbuergi
    added
    duplicateThis issue or pull request already exists
    on Sep 22, 2026

  10. rbuergi commented on Sep 22, 2026

    @rbuergi
    rbuergi
    on Sep 22, 2026
    Contributor

    Closing as a duplicate of #5009 — the same condition, re-reported from a pre-fix image. A duplicate disposition asserts nothing about the fix; #5009 stays open and closes only on production verification.

    Same condition. Every retained sample here is #5009's head-of-line shape exactly: 64 in flight, stream destinations queued 1, deepest per-destination queue 62, target cache/<meshId>, episode stamps at #1 that drain and recur (the two deepest-26-over-3 lines are the same backlog split three ways — ~InFlight/N, as the line itself says). The addresses recur across pods (cache/FtxSQnNgjkmJzr-09vwCJQ, cache/adQjx8eg8UyB5mWb_cRd5Q), which is #5151's diagnosed mechanism: cache/{meshId} is per PROCESS, so the queue that stacks is each pod's channel to a PEER pod's cache hub, one network round trip per frame, and this fleet has 8 pods where #5009 had 2 — more cross-silo channels, more of exactly this. Both incidents were re-addressed out of the same old category-wide fingerprint e4a97855ab595beb (= SHA256("MeshWeaver.Hosting.Orleans.RoutingGrain\n0\n")[..8], per Doc/Architecture/LogWatchTriage), which is why one condition holds two tickets.

    The fix is merged and not live. PR #5151 (merged 78bbf49e03, 2026-09-21T22:43Z) re-keys OrderedRouteDispatcher's FIFO from the destination address to the (destination, stream) pair, so independent streams stop waiting on each other. Three independent readings say no sample here came from an image carrying it:

    1. Timestamps. First seen 12:32Z, last seen 23:03Z on 2026-09-21 (the bot's fourth fold, 93 total) — the last 20 minutes are after the merge, none are after a roll.
    2. The line's own wording — this is the instrument, not the timestamps. fix(routing): the ordering FIFO's channel is (destination, stream), not the destination #5151 renamed the fields: a post-fix line reads ordered channels queued N over M stream destination(s), deepest per-channel queue D. All 93 folded occurrences print the pre-fix stream destinations queued …, deepest per-destination queue …. A post-fix(routing): the ordering FIFO's channel is (destination, stream), not the destination #5151 image cannot emit that sentence.
    3. Image provenance (Ops/Status/* on the control instance, sampled 2026-09-22T06:13Z, converged: true): memex-cloud 6/6 on 3.0.0-ci.9076 = commit 026442fff0 (fix(#4716): a submission inbox on a public partition — the grant shape both permission folds agree on, and a boot heal that stopped deleting it #5043); memex 2/2 on 3.0.0-ci.9106 = 8a52945608 (feat(comms): the Communication Hub — open vocabularies, durable dispatch chain, and the first enum conversion #5044). git merge-base --is-ancestor 78bbf49e03 <running> → NO for both; control: the memex commit IS an ancestor of the fix(routing): the ordering FIFO's channel is (destination, stream), not the destination #5151 merge, i.e. older.

    The pods list here spans BOTH portals' current generations (77494bdbcc is memex-cloud's, 7b4776b9f is memex's) plus two retired ones — the namespace: memex-cloud field is the first-seen namespace, not the population.

    Not folded, deliberately: nothing here carries #5018's deepest 0 load shape; that ticket is untouched by this and by #5151.

    What closes #5009 (and would re-open this question): after a roll carrying 78bbf49e03, a crossing that prints deepest per-channel queue ≥ 1 is frames of ONE stream stacking — a genuinely stuck leg, a different defect. A crossing printing many channels over few destinations with deepest 0 is load. A line still printing per-destination on a post-roll pod is the roll not having reached that pod.

  11. rbuergi
    closed this as duplicateon Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingduplicateThis issue or pull request already existssev:HBLOCKING: primary path broken, intermittent user-visible failure, or silently wrong results.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions