Repository navigation
RoutingGrain dispatch slots saturate with legs head-of-line blocked behind one shared-cache destination while the second pod starves on load #5009
Description
What is failing
The Orleans RoutingGrain dispatch-slot pool (64 slots) saturates repeatedly on both memex-portal pods, and the two pods show two different failures at the same log site:
- Pod s246c (head-of-line blockage): in every burst, ~62 of the 64 in-flight legs are queued behind a single destination —
cache/12xX8OXQIEmdwvf_ZZ8LjA, a shared node-cache address — withrouting pool subscribing 0andstream destinations queued 1. By the alert's own semantics, a deepest per-destination queue of 62 means legs are blocked behind one leg that is not terminating. - Pod ndnxt (load/starvation): the same threshold is crossed with
deepest per-destination queue 0andstream destinations queued 0— nothing is waiting on anything, so this is dispatch volume / CPU starvation on the silo, not a stuck delivery.
Rising episode stamps on both activations (#1→#3 on s246c, #6→#12 on ndnxt) mean each episode drained — slots are not leaking and the grain was not recycled. The problem is recurrence: the routing layer ran at its back-pressure ceiling for ~1.5 hours straight.
Probable cause
- s246c — medium confidence: a slow or stuck leg to the shared node-cache destination
cache/12xX8OXQIEmdwvf_ZZ8LjA. The same shape recurs in Ops logs as[STALE-CALLBACK] cache/…: N callback(s) pending > 30000msgoing back to 2026-08-12 on thememex-cloudcluster — deliveries to cache addresses that hang for >30s. A head leg to that destination stalling would queue every other cache-bound leg behind it and hold 62 of 64 slots. The evidence supports this; it does not prove it — the episode stamps show the blockage does eventually clear, so the leg is slow, not permanently dead. - ndnxt — lower confidence: pure dispatch-load starvation. The 2026-09-14 fix moved bulk imports/copies/moves off the routing switchboard precisely because heavy work competing with routing produced this symptom; worth checking whether the traffic running 19:07–20:34 today was another caller reintroduced on the switchboard, or simply legitimate volume the silo can't drain.
Impact
17 critical bursts over ~87 minutes (19:07–20:34 UTC) across both pods of the memex-cloud portal. Every message the portal routes passed through a grain repeatedly running at its back-pressure ceiling — user-facing page subscriptions and sync streams routed through these grains saw recurring head-of-line queuing (s246c) or slot exhaustion (ndnxt). Not a full outage — episodes drained — but a sustained degradation, not a blip.
Where to look
- The dispatch-slot pool and per-destination leg queues in
MeshWeaver.Hosting.Orleans.RoutingGrain— specifically why 62 legs can pile behind one destination before any bound kicks in. - The leg that dispatches to
cache/12xX8OXQIEmdwvf_ZZ8LjAon pod s246c, and the historical[STALE-CALLBACK] cache/…entries inOps/Logsfor the same cluster. - Action-Block Wedge Prevention — Invariant 3 covers the hub-level aggregate breaker; this incident is the routing-leg analogue (many legs, one blocked destination).
- Imports and copies no longer run on the part of the portal that routes traffic — the guard added here is the first thing to re-check for the ndnxt (load) shape.
- Routing back-pressure alerts now name what they actually saw — background on how to read these fields.
Note for the assignee: this log site reports genuinely different conditions (head-of-line blockage vs. load starvation) that the masking rule folds into one fingerprint — the two shapes above should be investigated as separate leads.
Evidence
| Fingerprint | 1ae8e45450e08823 |
| Category | MeshWeaver.Hosting.Orleans.RoutingGrain |
| Severity | Critical |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-69956b6dbc-ndnxt, memex-portal-deployment-69956b6dbc-s246c |
| Occurrences | 17 |
| First seen | 2026-09-20 19:07:47Z |
| Last seen | 2026-09-20 20:34:25Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.RoutingGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-20 20:07:47Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [9d926d64#1 started 2026-09-20T20:07:47.8062870Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/12xX8OXQIEmdwvf_ZZ8LjA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 20:07:47Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [bd087f9a#6 started 2026-09-20T20:07:47.8263262Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Ifrs17/PartnerRating — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 20:27:47Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [bd087f9a#8 started 2026-09-20T20:27:47.7964913Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Edu/Quiz — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 20:27:47Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [b39b1032#1 started 2026-09-20T20:27:47.7985847Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/12xX8OXQIEmdwvf_ZZ8LjA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 20:30:35Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [bd087f9a#9 started 2026-09-20T20:30:35.8246860Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Hosting/IssueClassificationRun — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 20:30:35Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [b39b1032#2 started 2026-09-20T20:30:35.8417081Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/12xX8OXQIEmdwvf_ZZ8LjA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 20:30:36Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [bd087f9a#10 started 2026-09-20T20:30:36.3321684Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 1. Latest dispatch target Reinsurance/AggregateSection — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 20:30:36Z memex-portal-deployment-69956b6dbc-s246c crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [b39b1032#3 started 2026-09-20T20:30:36.3622892Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 1, deepest per-destination queue 62, routing pool subscribing 0. Latest dispatch target cache/12xX8OXQIEmdwvf_ZZ8LjA — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 20:32:56Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [bd087f9a#11 started 2026-09-20T20:32:56.3249797Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 2. Latest dispatch target Ifrs17/LineOfBusiness — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
2026-09-20 20:34:25Z memex-portal-deployment-69956b6dbc-ndnxt crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
[ROUTE] Routing back-pressure [bd087f9a#12 started 2026-09-20T20:34:25.8222055Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 1. Latest dispatch target Edu/PromptCell — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/1ae8e45450e08823. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 21, 2026
Consolidation: this is the canonical ticket for the cache-destination head-of-line block
29 of the 87 bot-filed tickets opened 2026-09-19 → 2026-09-21 describe one defect: a route leg to a
cache/*pod-hub destination does not terminate, and because dispatches to one destination are serialized, the per-destination queue absorbs nearly the whole 64-slot in-flight budget. They are being closed as duplicates of this ticket. Everything below is evidence those tickets carried that this one did not — it is migrated here so nothing is lost with them.Closed into this ticket: #4897, #4899, #4900, #4904, #4908, #4989, #4992, #4993, #4994, #4995, #4996, #4998, #4999, #5000, #5002, #5003, #5004, #5005, #5006, #5007, #5010, #5012, #5014, #5029, #5031, #5032, #5061, #5063.
1. It is not one bad cache hub — twelve distinct destinations were measured
cache/12xX8OXQIEmdwvf_ZZ8LjAcache/PlOy4bo4wUGD_muJVcJzcgcache/YORFhwiyqEyTP9ESF-NUNgcache/pY9MmqY9YECUi-OLU-u1pAcache/Lmzw6ArfrECTSTdqx1hFTgcache/Xpi-G1Q-lU6s2Moll9rUcwcache/sJ0Xk1NSsEaHD_ONqJ314Qcache/vyK5n1IlsUqe7j8OHpW-Sgcache/_mPaH5nYqE2LNR0tiWR3DQcache/UQaJgLdynkKmdsldLQJQggcache/V9iMXHCKVkurS1KMrArQ3Q,cache/z_5Dny2pKkeS2yNS_gWx7Acache/fT_xtrn7fkukavHMBB4MqABoth production portals, every portal pod. So the fix is in the routing leg's bound, not in one destination hub.
2. The depth is NOT always 62 — it scales with the backlog
62is the common reading, not a signature: #5031/#5032 measured 16 and 13, #5029 measured 25, #5002 measured 14 and 62, #5063 measured 5, 36, 39 and 62 in one 53-second burst. #5063's reading of its own mixed depths is that the backlog was spread across several destinations rather than pinned on one — worth keeping as the counter-example to "one slow hub".3. It is not a leaked slot, in every longitudinal sample
Episode stamps advance on the same activation, which by the log site's own rule means each episode drained:
871d4cfbAdd entire company to Open SMC Teams #1 → Update Directory.Build.props #26 across 7.5 hours (the longest run measured), grain never recycled.d89f0147Add entire company to Open SMC Teams #1 → [Snyk] Upgrade axios from 0.25.0 to 0.27.2 #8 across 5 hours, 6 of 8 samples carrying the head-of-line signature oncache/UQaJgLdynkKmdsldLQJQgg.7fa5f98aamend nuget package info #29 → Cleanup #38 over ~2 hours.eb59bdbbTest #2→Clarify license options with ag grid #4 then Get rid of project dependencies in NotebookEditorControl #5→moved canEdit and canRun permissions to props #6,1c7f1383Add entire company to Open SMC Teams #1→Determine back-end contributors #3, i.e. the block cleared and re-formed 7 minutes later, same destination.Two samples do not settle it and should not be read as settled: #5003 and #5014 each captured a crossing (
d9e0b2de#3/#4,b5b0cd54#1) with no higher episode and no clear line inside the sample window. That excludes nothing; it is the one reading that would turn this into a slot-release bug.4. Periodicity, and the "dead pod hub" reading
#5000 is the sharpest single hypothesis and is worth reading in full: 8 crossings on a ~37-minute cadence, identical destination and identical depth 62 every time — which it reads as a periodic dispatcher re-triggering against a hub that is not there, rather than random load. It cites prior evidence: on 2026-09-14 the same namespace logged repeated
[ROUTE] Directed delivery to pod hub 'cache/…' was refused(Ops/Logs/memex-1789416031191996185-6b6b44-klrt7and neighbours). Refused ≠ terminated (#4900's phrasing) — a refused delivery may leave its leg hanging.5. Corroborating log family: stale cache callbacks
[STALE-CALLBACK] cache/…: N callback(s) pending > 30000msrecurs inOps/Logsback to 2026-08-12. #5061 names instances:cache/fT_xtrn7fkukavHMBB4MqAwith up to 6 pending,cache/zKwDrfWglUmNHD3114bDxA,cache/kiEWsIS0wkadPmkFkynoog. If a cache-hub callback can hang for >30 s, that is long enough to produce every reading above.6. Volume and simultaneity
Largest bursts: #5006 40 occurrences over 2 h across 3 pods; #4992 22 over 2 h; #4989 20 over 1.5 h; #5007 17 over 1 h; this ticket 17 over 87 min. #4900 measured both pods crossing within 11 ms of each other — one platform-wide event, not two coincidences. #4999 measured all four pods crossing inside the same 4-minute window.
7. Prior art the duplicates surfaced
sev:H, closed) — "RoutingGrain route dispatches hang at 64 in-flight when _Activity compilation stalls", the same log site. RoutingGrain dispatch slots saturate with non-terminating route legs (repeat of #1172/#5036) #5063 still names it as the primary ticket; it is closed, and the defect is live. Whatever closed it did not hold.Doc/Architecture/ActionBlockWedgePreventionInvariant 3 (the hub-level aggregate breaker — this is its routing-leg analogue), andDoc/WhatsNew/2026-09-14-imports-and-copies-no-longer-run-on-the-part-of-the-portal-that-routes.8. What none of the 29 establishes
Why the head leg does not terminate. The two candidates — a stuck cache-hub callback, and the unbounded pre-timeout ThreadPool wait that the slot is held across — are not separated by any sample in the set. That separation is the first task, not the fix.
Consolidated from the bot-filed incident backlog. Severity assigned per
Doc/Architecture/IssueTaxonomy: sev:H — an intermittent, user-visible failure of the primary path (everything the portal routes), live on both production portals and recurring hourly for at least two days, with episodes that degrade rather than wedge.rbuergi commentedon Sep 21, 2026
Consolidation 2026-09-21 — 19 tickets folded here
Folded as duplicates: #5091 #5092 #5094 #5095 #5127 #5129 #5130 #5131 #5132 #5133 #5134 #5137 #5138 #5139 #5141 #5142 #5144 #5145 #5147.
Grouped on the discriminator the log site itself defines, not on titles: a deepest per-destination queue ≥ 1 means legs are blocked behind a leg. Every ticket folded here measured a deepest reading of 23–62 of the 64-slot budget; the empty-queue (
deepest 0) shape is a different verdict at the same log site and is tracked separately at #5018, and the alerting-policy verdict at #5118. All three were being filed as the same-looking title.The finding worth acting on: six tickets, one stuck client
cache/wpWb1fSJPUacoVxY6LnltQis the named blocked destination in #5129, #5130, #5131, #5132 and #5133 (09:08Z–09:42Z, deepest 41/62/62/48/61) and its nodeops siblingportal/nodeops-wpWb1fSJPUacoVxY6LnltQin #5134. Six separately-filed incidents, one client. Two more client pairs show the samecache/<id>+portal/nodeops-<id>pattern:9VkLPkBrRU-rwTJ4WyrfiQ(#5137, #5139, #5145) andTmrXhZkCpE6r94nulQBzpQ(#5142, #5144).🔗 Cross-group link: that same client address,
cache/wpWb1fSJPUacoVxY6LnltQ, is the one whoseSubscribeRequests were discarded at 08:21Z by theHosting/TriageItemrecycle cascade — see #5110, folded into #4888. It began head-of-line blocking routing ~47 minutes later. Whether the dropped-subscription state is what wedges the client, or both are downstream of a third cause, is not established — but it connects the two largest groups in this backlog and is the most promising thread in the burst.Denominator
memexandmemex-cloudboth (#5138 and #5141 are memex-cloud), so the shape is not confined to one portal. #5134 is the cleanest slot-leak sample: episode7c973d7f#1is followed by neither a clear nor a higher episode, which by the log site's own rule means the leg never terminated. #5147 additionally lost a user-visible agent-thread message (UWDeepfield/IndustryNews/_Thread/…); that directed-delivery half is #2299.rbuergi commentedon Sep 21, 2026
Root cause: the ordering FIFO's key, not a slow leg — fix in #5151
Not a stuck or slow
cache/…leg. The ticket's medium-confidence hypothesis (a stalled leg to the shared node-cache destination, supported by the historical[STALE-CALLBACK] cache/…lines) is not what the code does. The mechanism is inOrderedRouteDispatcher, and the ticket's own "Where to look" asks the right question — why 62 legs can pile behind one destination before any bound kicks in.The queueing structure
OrderedRouteDispatcherserialises stream-routed legs one in flight per destination address, so the delta protocol's receive-side monotonicity guard stays sound (reorder two frames of one stream and the earlier one is discarded as stale forever, with nothing logged above Debug on either side).But that guard operates per mirror — per synchronization stream — and a stream-routed address is a multiplexer.
cache/{meshId}fronts onesync/{streamId}sub-hub per node the process mirrors, andDataExtensions.RouteStreamMessagereceives every frame at the cache hub's own address and forwards it internally byStreamId. So keying the FIFO on the address alone put a whole process's data-sync traffic on one channel with one in-flightIPodHubGrain.Delivergrain call — a directed call carrying the full delivery, which Orleans'JsonCodeccopies both ways.Where that destination is on another silo, the channel's service rate is one network round trip per frame. A busy portal's change fan-out exceeds it, and a single-server queue then grows.
The asymmetry in this ticket is the proof, not a second lead
The ticket reads the two pods as two leads. They are the two ends of one cross-silo channel:
cache/{meshId}is keyed by the mesh hub's id, which is per process, so each pod has its own cache address and its ownPodHubGrainactivation on its own silo.deepest 0.Which is why
s246cqueued 62 deep behindcache/12xX8OXQIEmdwvf_ZZ8LjAwhilendnxtshoweddeepest 0, and whygx6z6saw the same12xX8…address on 09-19: in both windows the queueing pod is the non-ndnxtpod of the day, i.e.12xX8…is the long-lived pod's cache hub and the queue is always the remote lane.Rising episode stamps on both activations are consistent with this and rule out the alternatives the line itself names: every episode drained, so no slot leaked and no leg was stuck. The 62 legs were independent streams with no ordering relationship, waiting on each other because the channel key could not tell them apart. The log line's rule (deepest ≥ 1 means legs are blocked behind a leg) was applied correctly and still produced the wrong conclusion, because the line is silent about why the legs share a queue.
Why #1172 (cited as primary) did not hold
It was never closed by a fix to this structure, and it predates the instrument that separates the two shapes.
Updatedno longer costs a storage query per routed message).dotnet-stack/EventPipe profiling on the pod reclassified it — "nothing is stuck. The silo is CPU-starved" — and a further comment corrected the ticket outright: 64 isRoutingGrain.SaturationThreshold, not Orleans'NonReentrancyQueueSize, and "nothing throttles, queues or refuses at 64".deepest per-destination queueto it.So #1172's accepted diagnosis is
deepest 0, i.e. #5018's shape. Every fix on it lowered the arrival rate or service time of a lane that was still one lane wide (#1179, #1341, #1358, and the seven-copies-per-timeout amplifier). A single-server queue whose arrival rate exceeds its service rate recurs at any offered load above that rate — reducing the load moves the threshold, it does not remove it. Both #5009 and #5018 inherited the same fingerprinte4a97855ab595beb, which is why "#1172 is primary" kept being cited for a shape it never covered. (#4903 was closed as a duplicate consolidated onto #4904; the shape itself was never worked.)Doc/Architecture/ATimedOutDeliveryIsStillHeldByTheCalleealready states this diagnosis in prose and marks it "Not addressed here." #5151 is that.The fix
The channel key becomes (destination, payload identity). The identity is
IDiagnosticKeyed.DiagnosticKey— the stream id for everyStreamMessage— whichMessageDelivery.Packagealready stamps on the envelope before erasing the payload type, so the routing grain reads it with one dictionary lookup of a string on the turn: no cast, nothing parsed. Same stream ⇒ same channel (order preserved exactly); different streams ⇒ different channels; no identity ⇒ the destination-wide channel, unchanged.No bound is raised: the routing
IIoPoolcap,SaturationThresholdandPodHubGrain's non-reentrancy are untouched. Controls measured on both sides — with the old key the two blocking tests fail and the 8 order/load/drain controls pass; with the new key all 10 pass.Not closing this on the merge
The defect is only observable under real cross-silo fan-out on a multi-pod portal. Verification, after the image is confirmed running per instance on the Fleet Console:
deepest per-channel queue 0withordered channels queued≈ the in-flight count over few destinations. A line still spellingdeepest per-destination queueis a pre-fix replica, not a result.deepest per-channel queue≥ ~10 on acache/…destination means frames of ONE stream are genuinely stacking up — a different, real defect this does not cover; re-open naming it.RoutingGrainbinds no NodeType, so no address needs an explicitrecycle— but the pods must be replaced for the new assembly to serve.test: convert OrderedRouteDispatcherTest's timeouts, and lower the ra…
rbuergi commentedon Sep 21, 2026
The fix is merged — and this issue stays OPEN until a roll verifies it
#5151merged tomainthrough the merge queue: the ordering FIFO's channel is(destination, stream), not the destination.🚨 Not closing this on the merge, and the reason is not caution. The defect is only observable
under real cross-silo fan-out on a multi-pod portal, so nothing in CI can show it gone.
RoutingGrainbinds no NodeType and needs no
recycle, but the pods must be REPLACED for the new assembly toserve — a roll ends the activations on the pods it replaces. Until an instance is confirmed running
the new image on the Fleet Console (
/Hosting/Console), the only thing established is that themechanism was identified and the controls hold.
What to read on the rolled portal, in order
[ROUTE] Routing back-pressurelines, if any appear, readdeepest per-channel queue 0withordered channels queued≈ the in-flight count over a smallnumber of destinations. Many channels over few destinations is load, and is the expected
post-fix shape for the traffic that used to read
deepest 62.deepest per-destination queueis a PRE-FIX REPLICA, not a result.The field was renamed, so the old spelling is how you tell which image produced a given sample —
which matters in any window that straddles the roll.
deepest per-channel queue≥ ~10 on acache/…destination means frames of ONEstream are genuinely stacking behind one another. That is a different, real defect this fix does
not cover — re-open naming it rather than reading it as "the fix did not work".
No increase in "awaiting first data" / "Building layout…" stuck views. A reordering regression logs
nothing above Debug on either side, so it has to be watched, not queried.
Where the reasoning lives now
The grouping of this issue against the other five in the cluster — and the finding that for four of the
six the causal arrow runs the other way (a silo that has lost its CPU or heap raises this gauge as
a symptom) — is recorded at
#5018 (comment).
Two things that bear on this issue specifically:
reading in the whole backlog shows episode stamps rising on the same activation:
871d4cfbAdd entire company to Open SMC Teams #1→Update Directory.Build.props #26over 7.5 h,
d89f0147Add entire company to Open SMC Teams #1→[Snyk] Upgrade axios from 0.25.0 to 0.27.2 #8 over 5 h,e9b212fbAdd entire company to Open SMC Teams #1→Clarify license options with ag grid #4,ca8c05b2Add entire company to Open SMC Teams #1→Determine back-end contributors #3 in 26 s. Slots arereleased and the grain is not recycled.
A separate change adds the oldest in-flight leg's age and label to this same line, so one sample
distinguishes load from a leaked slot without needing a second line to arrive. That is the reading
those three needed and could not have.
A named, unproven link worth re-checking after the roll
Every leg queued in
OrderedRouteDispatcherretains its wholeIMessageDelivery, so a 62-deeplane pinned 62 full payloads on top of Orleans' own
JsonCodeccopies. That makes this structure aplausible retention amplifier for the OOM bursts in #4824. Not established — I have no per-pod
back-pressure reading for
58dbb6d98d-8mswwin its 06:28–06:31Z window. Falsifier: if OOM burstsof that shape continue at the same rate after this rolls, lane retention was not a contributor.
fix(routing): the saturation report prints the gauge that can see the…
systemorph-com commentedon Sep 22, 2026
Still occurring: 1 more since the last update (97 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khldRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 1 more since the last update (98 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khldRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 2 more since the last update (100 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khldRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 4 more since the last update (104 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khldRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 3 more since the last update (107 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khldRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.31 remaining items
systemorph-com commentedon Sep 22, 2026
Still occurring: 2 more since the last update (175 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khld,memex-portal-deployment-77494bdbcc-6sv8l,memex-portal-deployment-77494bdbcc-kkjld,memex-portal-deployment-6466cf967f-4phpq,memex-portal-deployment-6466cf967f-gjzpf,memex-portal-deployment-77494bdbcc-5djgrRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 2 more since the last update (175 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khld,memex-portal-deployment-77494bdbcc-6sv8l,memex-portal-deployment-77494bdbcc-kkjld,memex-portal-deployment-6466cf967f-4phpq,memex-portal-deployment-6466cf967f-gjzpf,memex-portal-deployment-77494bdbcc-5djgrRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 3 more since the last update (178 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khld,memex-portal-deployment-77494bdbcc-6sv8l,memex-portal-deployment-77494bdbcc-kkjld,memex-portal-deployment-6466cf967f-4phpq,memex-portal-deployment-6466cf967f-gjzpf,memex-portal-deployment-77494bdbcc-5djgrRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 2 more since the last update (180 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khld,memex-portal-deployment-77494bdbcc-6sv8l,memex-portal-deployment-77494bdbcc-kkjld,memex-portal-deployment-6466cf967f-4phpq,memex-portal-deployment-6466cf967f-gjzpf,memex-portal-deployment-77494bdbcc-5djgrRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 1 more since the last update (181 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khld,memex-portal-deployment-77494bdbcc-6sv8l,memex-portal-deployment-77494bdbcc-kkjld,memex-portal-deployment-6466cf967f-4phpq,memex-portal-deployment-6466cf967f-gjzpf,memex-portal-deployment-77494bdbcc-5djgrRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 1 more since the last update (182 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khld,memex-portal-deployment-77494bdbcc-6sv8l,memex-portal-deployment-77494bdbcc-kkjld,memex-portal-deployment-6466cf967f-4phpq,memex-portal-deployment-6466cf967f-gjzpf,memex-portal-deployment-77494bdbcc-5djgrRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 1 more since the last update (183 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khld,memex-portal-deployment-77494bdbcc-6sv8l,memex-portal-deployment-77494bdbcc-kkjld,memex-portal-deployment-6466cf967f-4phpq,memex-portal-deployment-6466cf967f-gjzpf,memex-portal-deployment-77494bdbcc-5djgrRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 3 more since the last update (186 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khld,memex-portal-deployment-77494bdbcc-6sv8l,memex-portal-deployment-77494bdbcc-kkjld,memex-portal-deployment-6466cf967f-4phpq,memex-portal-deployment-6466cf967f-gjzpf,memex-portal-deployment-77494bdbcc-5djgrRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 2 more since the last update (188 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khld,memex-portal-deployment-77494bdbcc-6sv8l,memex-portal-deployment-77494bdbcc-kkjld,memex-portal-deployment-6466cf967f-4phpq,memex-portal-deployment-6466cf967f-gjzpf,memex-portal-deployment-77494bdbcc-5djgrRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 1 more since the last update (189 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khld,memex-portal-deployment-77494bdbcc-6sv8l,memex-portal-deployment-77494bdbcc-kkjld,memex-portal-deployment-6466cf967f-4phpq,memex-portal-deployment-6466cf967f-gjzpf,memex-portal-deployment-77494bdbcc-5djgrRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 2 more since the last update (191 total).
memex-portal-deployment-77494bdbcc-7ndrn,memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-7fc5f4874b-9zwrg,memex-portal-deployment-77494bdbcc-bphfb,memex-portal-deployment-8d98c8fc6-w6clj,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-7b4776b9f-mqfr6,memex-portal-deployment-7b4776b9f-8khld,memex-portal-deployment-77494bdbcc-6sv8l,memex-portal-deployment-77494bdbcc-kkjld,memex-portal-deployment-6466cf967f-4phpq,memex-portal-deployment-6466cf967f-gjzpf,memex-portal-deployment-77494bdbcc-5djgr,memex-portal-deployment-77494bdbcc-rg99vRecent log lines
Re-addressed by the current identity function: this incident inherited
e4a97855ab595beb(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 22, 2026
#5261 is not a recurrence of #5009 — and it is #5009's first post-roll POSITIVE reading
#5261's body offers that it "may simply be a production recurrence of #5009". Its own log text refutes that, using the discriminator #5009 comment 3 fixed in advance: "a line still spelling
deepest per-destination queueis a PRE-FIX REPLICA, not a result" — #5151 renamed the field.ordered channels queued 0 over 0 stream destination(s), deepest per-channel queue 0. Incident 2026-09-22T17:45:19Z, podmemex-portal-deployment-5c99d79997-xs87x, namespacememex.per-destination. RoutingGrain dispatch slots saturate with legs head-of-line blocked behind one shared-cache destination while the second pod starves on load #5009 comment 22 records that an incident-store search forper-channel queue/ordered channels queuedfound no hit anywhere. RoutingGrain routing pool exhausted by slow dispatch legs — 51 of 64 in-flight dispatches waiting on a pool slot, oldest leg 26s #5261 is 1 of 1 carrying the post-fix spelling.So on that replica the head-of-line queueing is gone: 0 channels queued over 0 destinations, deepest 0. That is the positive signal #5151 has been missing, and it is the first one on file.
And in-flight still reached 64 anyway — for a different reason. #5261's own gauges:
waiting for a pool slot 51,subscribing 0, deepest 0, oldest leg 26,372 ms, that leg's targetOps/Logs/memex-cloud-…-v7q45. By the log line's own rubric ("High [waiting] while subscribing is below the pool's cap is a THREAD shortage") this reads as the unbounded pre-subscribe wait the slot is held across — not a too-coarse channel key.🚨 So deduping #5261 into #5009 would destroy the only post-roll positive sample on file. They are two defects that raise the same gauge, which is exactly the trap of reading a symptom gauge as an identity.
Not excluded, and worth the next session's first hour: a slot that is never decremented. Core #5281 (
f2f00c9db4, merged 18:35:15Z) fixed precisely the class "a pooled leaf holds a permit across work that cannot observe its token" — inOrleansRoutingService.cs, the same file family — so that shape was live today rather than speculative. It is not this fix: it touches the teardown unsubscribe leaf, not the routing dispatch leg or the pool's slot accounting, and it merged 50 minutes after the #5261 incident, so that pod cannot have carried it. The place to start is its new guard,test/MeshWeaver.Documentation.Test/PooledLeafObservesItsTokenGuard.cs— does its sweep cover the routing dispatch leg?What is NOT established here: none of this was read from code. The channel key's construction, the pool's cap, and whether any path fails to release a slot are all unverified — the above is the issue record plus #5261's own instrument reading, with no control run and no two-sided verification. #5037 and #2299 were not examined at all.
🤖 Generated with Claude Code
rbuergi commentedon Sep 22, 2026
A second post-fix reading, on the OTHER portal — and it is not the falsifier
The 19:06Z comment counts #5261 as "1 of 1 carrying the post-fix spelling". There is a second one on file, sitting in the evidence of #5271 (the incident-filer ticket), and it is on memex-cloud, so between them both portals now have a post-#5151 sample:
98112c599c16b3ae)memex/5c99d79997-xs87xdispatch → Ops/Logs/…76dea3f1ce951aa2)memex-cloud/5c89c7f5f9-sddb5stream-routed → portal/egXML8v7Y…Both spell
ordered channels queued … deepest per-channel queue, the field #5151 introduced, so both are post-fix replicas by this thread's own discriminator.Reading the memex-cloud one against the falsifier this thread fixed in advance ("
deepest per-channel queue≥ ~10 on acache/…destination means frames of ONE stream are genuinely stacking up"): the depth is 12, but the channel is on aportal/…circuit address, notcache/…, the backlog is spread over 8 channels on 3 destinations, and the oldest leg is 84 ms — i.e. 12 frames of one circuit's stream queued behind each other and draining in well under a tenth of a second. By the line's own rubric that is load being absorbed, not a stream that is not draining. It is not the falsifier; I am recording it because a depth of 12 would otherwise read as one at a glance.And #5261's crossing is the other gauge entirely (
waiting for a pool slot 51,subscribing 0, deepest 0) — the pre-subscribe ThreadPool wait, tracked on #5261, not a channel-key defect.Remaining before this can close — unchanged from 22:44Z: the counter-check (no rise in "awaiting first data" / "Building layout…" stuck views after the roll), which is silent by construction and has to be watched, not queried. I have not done it and cannot from here.
What I did NOT establish: both readings are single crossings; I read no episode follow-up, ran no
Logsquery, and did not confirm pod5c89c7f5f9-sddb5's image other than by the field spelling.rbuergi commentedon Sep 22, 2026
Closing as a duplicate of #5118 — the fingerprint-split flood.
Why this is a duplicate. Every one of these comes from the same log site —
[ROUTE] Routing back-pressure, categoryMeshWeaver.Hosting.Orleans.RoutingGrain— split into one issue per grain activation. The 8-hex activation id survived normalization (a0e2984b→a0e2{n}b):HexBlobrequired 16+ chars, andNumber()then masked only the digit-run preceded by a digit, leaving a per-activation residue in the hash.Measured over 101 verbatim bursts drawn from 29 issues, covering 42 activation ids: the old parser yields 41 distinct fingerprints, the current parser yields 1.
The fix is already merged in MeshWeaver.Plugins — #2165 (
MixedHexId), #2167, #2168 (positional episode/target masking) and #2218 (the portal re-parses raw samples, instead of hashing text a stale watcher already masked).It is not live yet. The control instance is sealed on an older set because the Plugins Portal image rebuild workflow fails at its GitHub App token step —
actions: writeis not granted to the installation — so core's rebuild is never dispatched. Until that is granted, new split issues will keep arriving, and an issue whose activation is still alive may reopen here.Not closed by this. The underlying routing condition — cache-destination head-of-line blocking versus ThreadPool starvation — is a real defect and stays open in #5118. The normaliser itself is tracked in #5270 and #5271.