RoutingGrain back-pressure episodes on memex-portal pods — silo thread-pool saturation and a head-of-line block behind cache/* destinations

What is failing

The Orleans RoutingGrain on the memex-portal deployment repeatedly hit its 64-dispatch in-flight reporting threshold (3 crossings in under half a second, on both portal pods). This is not a routing defect in itself: the sample set shows two distinct shapes — on pod gx6z6, a genuine head-of-line blockage with a deepest per-destination queue of 62 behind a cache/* destination, and on pod ndnxt a pure-load crossing with all queues empty (deepest 0). The episode stamps on the gx6z6 samples (#1 then #2 on the same activation within 340 ms) show episode #1 drained and re-crossed, so nothing appears permanently stuck.

Probable cause

Evidence points at thread-pool / CPU starvation on the silos rather than a stuck leg: the alert text itself notes a slot is held from dispatch through an unbounded wait for a ThreadPool thread, so a starved silo raises this with nothing blocked. This incident is also one of dozens of independent back-pressure fingerprints raised across the fleet today (2026-09-19, from ~11:41 to ~18:48), alongside concurrent LocalSiloHealthMonitor "Thread Pool execution stalled" incidents — consistent with periodic silo saturation. Confidence: moderate — cause is saturation, but the source of the load (a burst workload vs. an undersized thread pool vs. the 62-deep cache/* head-of-line consumer) is not determinable from these samples alone.

Impact

Brief, self-draining critical alerts (episodes cleared in under a second) on both memex-portal pods, but recurring in waves all day. No evidence yet of a leaked slot (no episode that never drained); the real user-facing risk is latency for deliveries queued behind the cache/* destination during the gx6z6 episodes.

Where to look

Start from MeshWeaver.Hosting.Orleans.RoutingGrain in the MeshWeaver.Hosting.Orleans routing layer (the [ROUTE] back-pressure log site).
Investigate why the silo's ThreadPool is starved during these windows — correlate with the LocalSiloHealthMonitor thread-pool-stall incidents from the same period.
For the gx6z6 shape, look at what consumes deliveries to cache/* destinations (a slow leg there head-of-line-blocks 62 queued legs).
The 2026-08-12 change note (Doc/WhatsNew/2026-08-12-routing-alerts-name-what-they-actually-saw) documents the semantics of these fields — the count is threshold-crossing, not a diagnosis, and the named target address is not a diagnosis either.

Known-related incidents: many sibling [ROUTE] Routing back-pressure fingerprints were filed today (e.g. Admin/_LogIncident/2ffa882b43e2b504, 155d7ffcac9123d3, 26dfad998e9f5893, 226541f4d6883bb2) — this looks like the same underlying saturation fanning out per activation id; a human may want to consolidate them under one investigation.

Evidence


Fingerprint	`17d5eca3379f3f76`
Category	`MeshWeaver.Hosting.Orleans.RoutingGrain`
Severity	Critical
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-69956b6dbc-gx6z6`, `memex-portal-deployment-69956b6dbc-ndnxt`
Occurrences	3
First seen	2026-09-19 18:47:47Z
Last seen	2026-09-19 18:47:47Z

Recent log lines

2026-09-19 18:47:47Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [b3fc6985#1 started 2026-09-19T18:47:47.4705808Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/YORFhwiyqEyTP9ESF-NUNg — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 18:47:47Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [f86b4565#9 started 2026-09-19T18:47:47.8039272Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 1. Latest dispatch target GoogleMaps/Gallery — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 18:47:47Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [b3fc6985#2 started 2026-09-19T18:47:47.8122938Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/12xX8OXQIEmdwvf_ZZ8LjA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/17d5eca3379f3f76. Recurrences are folded into this issue rather than opening new ones.}

Consolidated onto #4904 — the `deepest 62` reading of one back-pressure condition

[ROUTE] Routing back-pressure from MeshWeaver.Hosting.Orleans.RoutingGrain has been filed many
times over today. The log line carries its own discriminator:

A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one
destination); 0 means nothing is waiting on anything, so read it as load.

This ticket's samples read stream destinations queued 1, deepest per-destination queue 62 —
the head-of-line reading — so it belongs with #4904, which reports the same cache address, cache/YORFhwiyqEyTP9ESF-NUNg, in every sample.

Why one condition keeps producing separate tickets, and where that stands. The message embeds the
reporting activation's identity — [b3fc6985#1 started …]. The bot fingerprints on the normalized
message, and the hex-masking rule that ran on the image which filed this ticket required 16 or
more hex characters, so an 8-character activation id survived normalization and each activation
forked its own fingerprint.

That is already fixed in code: MixedHexId in
MeshWeaver.Plugins/src/MeshWeaver.Observability.Contract/LogLineParser.cs (Plugins#2164 / #2165) now
masks an 8+ hex token that carries both a digit and an a–f letter — b3fc6985 qualifies. The fix is
on Plugins main and is not running: every portal deploy is blocked by
Systemorph/Memex#437 (the portal ingress refuses a tlsSecret with no issuer, in force on core
main since 2026-09-19T08:36:24Z), so the filer in production is still the pre-fix one and will keep
forking this line until an image carrying the fix rolls. Two narrower residual masking gaps are
tracked deliberately as Plugins#2166.

Closing here as a duplicate. The condition itself is not dismissed — it is tracked on #4904.

Reopened — this fault is happening again.

Still occurring: 1 more since the last update (4 total).

Last seen: 2026-09-19 19:19:19Z
Pods: memex-portal-deployment-69956b6dbc-gx6z6, memex-portal-deployment-69956b6dbc-ndnxt

Recent log lines

2026-09-19 18:47:47Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [f86b4565#9 started 2026-09-19T18:47:47.8039272Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 1. Latest dispatch target GoogleMaps/Gallery — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 18:47:47Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [b3fc6985#2 started 2026-09-19T18:47:47.8122938Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/12xX8OXQIEmdwvf_ZZ8LjA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-19 19:19:19Z memex-portal-deployment-69956b6dbc-gx6z6 crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
      [ROUTE] Routing back-pressure [b3fc6985#9 started 2026-09-19T19:19:19.7188581Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Portfolio/Reporting — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.

_{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Re-closing — the reopen was the filer discarding the duplicate verdict, and the cause of that is now fixed for this fingerprint

This ticket was closed as a duplicate of #4904 at 19:06:22Z, and the bot reopened it at 19:20:26Z with
"Reopened — this fault is happening again." The fault is happening again — that part is true — but
it is tracked on #4904, and the reopen is not evidence against that.

LogIncidentFiler.Reopen decides on two things only: the issue reads Closed, and the newest occurrence
postdates closed_at. It never reads the issue's state_reason. For a fault that fires every ~20
minutes, a duplicate closure is therefore guaranteed to be undone within the hour — which is what
happened here, 14 minutes after the close.

What has changed since: the incident node Admin/_LogIncident/17d5eca3379f3f76 now points at #4904 rather than at
this ticket —

 "repository": "Systemorph/MeshWeaver",
-"issueNumber": 4913,
-"issueUrl": "https://github.com/Systemorph/MeshWeaver/issues/4913"
+"issueNumber": 4904,
+"issueUrl": "https://github.com/Systemorph/MeshWeaver/issues/4904"

so the next recurrence of 17d5eca3379f3f76 comments on #4904, and this ticket is no longer in the reopen path. The
underlying filer behaviour is filed as Systemorph/MeshWeaver.Plugins#2177, with this reopen as its
measured example.

The consolidation itself is unchanged and is in the comment above: this ticket's samples read
deepest per-destination queue 62 against cache/YORFhwiyqEyTP9ESF-NUNg, the head-of-line reading that
#4904 owns.

Navigation Menu

Repository navigation

RoutingGrain back-pressure episodes on memex-portal pods — silo thread-pool saturation and a head-of-line block behind cache/* destinations #4913

Evidence

Consolidated onto #4904 — the `deepest 62` reading of one back-pressure condition

Re-closing — the reopen was the filer discarding the duplicate verdict, and the cause of that is now fixed for this fingerprint

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

RoutingGrain back-pressure episodes on memex-portal pods — silo thread-pool saturation and a head-of-line block behind cache/* destinations #4913

Description

Evidence

Activity

rbuergi commented on Sep 19, 2026

Consolidated onto #4904 — the deepest 62 reading of one back-pressure condition

systemorph-com commented on Sep 19, 2026

rbuergi commented on Sep 19, 2026

Re-closing — the reopen was the filer discarding the duplicate verdict, and the cause of that is now fixed for this fingerprint

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Consolidated onto #4904 — the `deepest 62` reading of one back-pressure condition