RoutingGrain route dispatches hang at 64 in-flight when _Activity compilation stalls #1172

Activity

1. rbuergi

   closed this as completedin #1179on Aug 10, 2026
2. rbuergi

   reopened this on Aug 12, 2026
3. rbuergi commented on Aug 12, 2026

   

   rbuergi

   on Aug 12, 2026

   Contributor

   More actions

   Reopening — the signature is live again, in prod and in a 2-vCPU CI container, on 2026-08-12

   #1179 fixed a real amplifier (an in-place Updated no longer costs the router a storage query per message). It did not end this.

   Two environments, same day, same two lines:

   • prod memex-cloud, pod memex-portal-deployment-bdc9f8dc4-fdkgw, 11:44:34Z — .NET Thread Pool is exhibiting delays of 10.7428933s (issue A write's cost is nine-tenths PROPAGATION: owner merge, persistence, and per-subscriber fan-out each re-materialise the whole document #1284).
   • Systemorph/education e2e, main run 31594666242, 12:24Z, disposable portal in docker-compose on a 2-vCPU runner — .NET Thread Pool is exhibiting delays of 26.4076093s, degradation 8/8, alongside this issue's exact critical:

   crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
     [ROUTE] 64 route dispatches are in flight on this silo and not terminating
     (latest target Harness/_Activity/import-005c5c6b82cf5c03-20260812120954005-215492ad).
   

   Always exactly 64, always an _Activity/compile / _Activity/import target — Store/Coupon/_Activity/compile-state, Publish/Slide/_Activity/compile-state, Store/Catalog/_Activity/compile-…. Every education shard, every run today. That it reproduces on a 2-vCPU container and on multi-core AKS pods with server GC rules out both environments as the cause.

   The part #1179's analysis got wrong, with evidence

   #1179 concluded "routes to a pending activation do NOT hold dispatch slots … the slot-holder is the per-message resolution." The e2e traces say the routing leg is fast and the destination hub's turn is what is held:

   [STALE-CALLBACK] SubscribeRequest@e2e-get-risktransfer/RiskTransfer (48338ms)
      → ROUTED onTarget=True state=Submitted (+23ms)
      → HANDLER_ENTER (+48339ms)   ⇒ a handler was entered and no reply/completion/fault since
   
   warn: Orleans.Runtime.Scheduler.WorkItemGroup[101215]
     Task … [Activation: …/Source/L7ExtendGreen@… GrainType=MeshWeaver.Hosting.Orleans.MessageHubGrain …]
     took elapsed time 0:02:17.37 for execution, which is longer than 00:00:01. Running on thread 32
   

   Routed in 23 ms; handler entered 48 s later. A single grain turn running 2 m 17 s on one thread pins a ThreadPool thread, and the pool injects replacements at ~1/s — which is why Orleans' own timers then slip (UpdateIAmAlive … 00:01:34.6 ago), IMemoryStreamQueueGrain.Dequeue times out at 30 s on four queues at once, and the health monitor reports the aggregate as a thread-pool delay. The 64-in-flight critical is the reporter at the end of that chain, not its origin.

   So the open question is narrower than it was: what runs for minutes synchronously on a node hub's grain turn during a compile/import?

   Next diagnostic

   Orleans names it for free — WorkItemGroup[101215] … took elapsed time carries the activation address. Collect all of those from one bad window (prod Loki around 11:44:34Z on bdc9f8dc4-fdkgw; education e2e needs its docker compose logs --tail 300 replaced with a full timestamped artifact — the onset is currently outside the captured window) and group by activation. Full write-up in #1284.
4. rbuergi commented on Aug 12, 2026

   

   rbuergi

   on Aug 12, 2026

   Contributor

   More actions

   Root identified by live profiling of prod memex-cloud — and it is not a deadlock

   Profiled the running pod today (memex-portal-deployment-7574f9474c-zvqzn) with dotnet-stack / dotnet-counters via an ephemeral --target container. Both the analysis in #1179 and the "stuck / hung delivery leg" framing in this issue's body are inverted: nothing is stuck. The silo is CPU-starved, and the 64-in-flight cap is the reporter at the end of that chain.

   The culprit: unbounded synchronous Rx recursion on a hub turn

   Five threads, sampled simultaneously, each 62–97 frames deep, all rooted in MessageService.DrainOne() — i.e. five hub action-block turns at once, each burning CPU_TIME and contending on Monitor.Enter_Slowpath:

   System.Threading.Monitor.Enter_Slowpath(object)
   System.Reactive!…ObservableImpl.SelectMany`2+ObservableSelector+_.OnNext(!0)
   System.Reactive!…BasicProducer`1.SubscribeRaw(IObserver`1<!0>,bool)
     … × 33 nested SelectMany → SubscribeRaw levels …
   System.Reactive!…ObservableImpl.Select`2+Selector.Run(_<!0,!1>)
   System.Reactive!…ObservableImpl.Catch`2.Run(_<!0,!1>)
   System.Reactive!…ObservableImpl.Defer`1+_.Run()
   System.Reactive!…ObservableImpl.Do`1+Actions.Run(_<!0>)
   System.Reactive!…ObservableImpl.Catch`2.Run(_<!0,!1>)
   System.Reactive!…ObservableImpl.Finally`1+_.Run(IObservable`1<!0>)
   System.Reactive!…ObservableImpl.Defer`1+_.Run()
   System.Reactive!…Concurrency.CurrentThreadScheduler.Schedule(…)      ← runs INLINE on the turn
   System.Reactive!…Producer`2.SubscribeRaw(IObserver`1<!0>,bool)
   MeshWeaver.Messaging.Hub!MeshWeaver.Messaging.MessageService.DrainOne()
   System.Threading.ThreadPoolWorkQueue.Dispatch()
   

   Each SelectMany.OnNext synchronously subscribes the next producer, and CurrentThreadScheduler runs that trampoline inline on the calling thread rather than yielding — so one DrainOne() turn becomes a deep, unbounded, synchronous recursion that both pegs a core and allocates an observer/disposable per level.

   The measured consequences

   Pod limits are cpu: 6 / memory: 28Gi. From /sys/fs/cgroup/cpu.stat over an 11.6-minute window:

   CFS periods throttled	3583 / 6986 = 51%
   Throttled time	1490 s of 699 s wall
   CPU consumed	4.28 cores avg, user-mode (user_usec 2992 s vs system_usec 26 s)
   Thread attribution	.NET TP Worker ~6.4 cores + GC (BGC+Server GC) ~1.4 cores ⇒ ~7.8 cores demanded against a 6-core quota
   Working set	17–19 GiB steady

   Why this looks like "stuck legs" and "thread-pool starvation"

   Under CFS throttling the whole process is frozen for the remainder of each period. That is directly visible in the Orleans logs: at 14:32:27.275, 16 different activations on 16 different threads each reported an identical took elapsed time 4.2s, all within 0.04 s of one another —

   14:32:27.275  4.2s thr=60  messagehub/MeshWeaver/_Activity/4ad2a7c6
   14:32:27.275  4.2s thr=54  messagehub/DataModeling/09-Charts/Solution
   14:32:27.275  4.2s thr=58  messagehub/Skill/_Activity/import-eec5f76…
   14:32:27.275  4.2s thr=50  messagehub/DataModeling/09-Charts/Solution/GrowthSeries
   14:32:27.275  4.2s thr=35  messagehub/Skill/markdown
   …10 more, same millisecond, same duration
   

   Identical durations across unrelated activations is the signature of a process-wide stall, not of any one handler holding a turn. Grouping all 462 WorkItemGroup … took elapsed time lines from the day by activation shows the same thing: 251 of them fall in a tight 4.0–5.0 s band spread over 82 distinct activations, in synchronized bursts.

   So the causal order is:

   recursive Rx on DrainOne burns cores → cgroup quota exhausted → CFS freezes the process → Orleans turns, timers and IMemoryStreamQueueGrain.Dequeue all slip together → LocalSiloHealthMonitor reports a ThreadPool delay (#1284) → routing dispatches accumulate to the 64 cap (this issue).

   The _Activity/compile and _Activity/import targets are not special — they are simply the addresses with the most traffic when the freeze lands.

   Corrections to the record

   • An in-place update no longer costs the router a storage query per message #1179's "per-message path resolution is the slot-holder" is wrong. Routing is fast (ROUTED +23ms); the cost is inside the destination turn.
   • A write's cost is nine-tenths PROPAGATION: owner merge, persistence, and per-subscriber fan-out each re-materialise the whole document #1284 has it backwards. Its thread-pool delay is not secondary to a routing backlog; the routing backlog and the thread-pool delay are both secondary to CPU starvation.
   • Not a leak. .NET File Watch threads sat at exactly 446 across a 17-minute re-sample (446 inotify fds / 1626 watches) and working set was flat at 17–19 GiB — steady state, not growth. I nearly "fixed" a non-bug here.
   • Same root as Systemorph/education#77. That suite runs the portal on a 2-vCPU runner, where this recursion is far more punishing — which is exactly why it is duration-shaped (the 8–13 min single-course job passes on every run; the 20–40 min shards fail).

   Still open

   Which SelectMany chain recurses. Every frame between DrainOne and the recursion is shared-generic System.__Canon, so no MeshWeaver lambda is named in the sampled stack. The operator signature at the base — Defer → Finally → Defer → Catch → Do → Catch → Defer → Select, then ~20 Producer.SubscribeRaw, then the 33× SelectMany recursion — should identify it from the delivery-pipeline composition. An allocation-profiled or symbol-resolved capture would name it outright.

   _{Method note: the portal image carries no SDK, so the tools were injected with kubectl debug --target=memex-portal --profile=sysadmin --image=mcr.microsoft.com/dotnet/sdk:10.0, linking the target's /proc/1/root/tmp/dotnet-diagnostic-1-*-socket. This leaves exited ephemeral containers mwprof…mwprof5 on that pod; they consume nothing and disappear with the next roll.}
5. rbuergi commented on Aug 12, 2026

   

   rbuergi

   on Aug 12, 2026

   Contributor

   More actions

   Symbol-resolved capture: the chain is named — JSON patch construction on the hub turn

   Follow-up to the profile above. 75 s EventPipe capture (Microsoft-DotNETCore-SampleProfiler + exception events) on the same live pod, dotnet-trace report topN.

   🚨 Read this before believing the top line

   The #1 exclusive entry is FileSystemWatcher+RunningInstance.TryReadEvent at 92.97% — and it is an artifact, not a finding. Microsoft-DotNETCore-SampleProfiler samples thread-time, not CPU-time, so 446 of the process's 488 threads sitting blocked in an inotify read() dominate the sample count: 446/488 = 91.4% ≈ 92.97%. The authoritative CPU attribution is the /proc/<tid>/stat delta, which puts those same 446 threads at 1 jiffy total.

   Anyone who runs this capture and "fixes FileSystemWatcher" will be optimizing idle threads. (Note cpu-sampling is collect-linux-only in dotnet-trace 9; there is no in-container kernel CPU profile.)

   The named driver chain

   Discounting the blocked-thread noise, the inclusive tree resolves the System.__Canon frames into real MeshWeaver methods:

   ThreadPoolWorkQueue.Dispatch                                   1.99%
   └─ MessageService.DrainOne()                                   1.86%   ← the hub turn
      └─ CurrentThreadScheduler.Schedule → Rx fold                1.74%
         └─ <WithHandler>b__0(IMessageHub, …)                     1.70%
            └─ SetCurrent(IMessageHub, …)                         1.17%
               └─ <ConfigureSynchronizationHub>b__67_8(…)         1.10%
                  └─ <ToDataChanged>b__0(ChangeItem`1<T>)         1.14%
                     └─ JsonSynchronizationStream.ToJsonPatch     0.82%
                        └─ …CreateSingleObjectPatch (DisplayClass31_0)  0.82%
                           ├─ ObjectPolymorphicConverter.Write    0.60%
                           ├─ PatchExtensions.DiffObject          0.23%
                           └─ ApplyPatchWithCorrectUnescaping     0.45%
      └─ UpdateStream(IMessageDelivery`1<T>)                      0.51%
   NpgsqlConnector.ReadMessageLong                                0.62%
   

   ~44% of all hub-turn CPU (0.82 of 1.86) is JSON patch construction in JsonSynchronizationStream, and the JSON leaves (Utf8JsonReader.ConsumeStringAndValidate, TranscodeHelper, WriteStringEscapeValue, JsonDocument.Parse/ParseValue/WriteComplexElement, JsonObject.InitializeDictionary, Buffer.MemmoveInternal) together are the largest block of exclusive time among non-idle frames. That also accounts for the ~1.4 cores of continuous GC: this path allocates a JsonDocument/JsonObject graph per write.

   So the earlier finding stands and is now specific: the recursion depth is the rule chain, not runaway recursion — MessageHub.HandleMessageAsync folds one nested SelectMany per rule (MessageHub.cs:695-697, guarded only at > 500), which is why depth measured 33–51 and varied per hub. Every rule costs a subscribe level on every message even when it doesn't match. The work inside that fold is dominated by serializing and diffing node content into a JSON merge patch, synchronously on the turn.

   Why this is expensive in practice, not just in theory

   Patch cost scales with node content size, and several writers grow their content monotonically while writing repeatedly. Directly observed on this pod: MeshWeaver/_Activity/import-1d1a65773fb0c6b6-… went Version 587 → 599 in 30 seconds and was still writing 10 minutes after the import began, with MergeGuard refusals and a MonotonicWriteGuard CONFLICT on the same node. NodeTypeCompilationActivity.AppendLog re-serialises the whole ActivityLog.Messages list per progress line — O(n²) in bytes over an import.

   (I dismissed AppendLog as too small in my earlier comment. Half-right: it is not the burner on its own, but the patch path it feeds is the largest named leaf, so that dismissal was too strong.)

   What I am deliberately NOT doing

   Not proposing an edit. The sanctioned remedy for synchronous work on an action block is IIoPool, but patch construction produces the patch that must be posted in order, so moving it off the turn is a semantics change, not a mechanical hoist — and the alternative reading (reduce patch volume/size rather than relocate it) is at least as well supported by the version-churn evidence. Picking between those without measurement is exactly the guess this investigation has avoided twice already.

   What would settle it

   Patch volume attributed by node path — bytes and count of CreateSingleObjectPatch per target — which separates "the patch path is inherently too slow" from "a few writers emit far too many, far too large patches". Cheapest form: a counter on JsonSynchronizationStream.ToJsonPatch keyed by stream/node path, or an allocation-profiled capture (gc-verbose) attributing JsonDocument/JsonObject allocation to call sites. Either is ~15 minutes on this pod using the kubectl debug recipe above, and unlike the stack sample it does not depend on catching a burst.
6. rbuergi commented on Aug 12, 2026

   

   rbuergi

   on Aug 12, 2026

   Contributor

   More actions

   Patch-volume attribution: measured. It is _Activity/import-*, and the growth curve rises.

   Measured directly in Postgres rather than by instrumenting code: version is the write count, and because every write re-serialises the whole node, version × content_bytes is a sound estimator of total bytes serialised per path. Aggregated over every *.activities table in the database.

   Q1 — which paths produce the most writes

   schema	path	writes	content now
   linkedin	LinkedIn/_Activity/7a9d90de…	10,011	1.8 kB
   agenticengineering	AgenticEngineering/_Activity/import-69f6248b…	5,239	141 kB
   uwdeepfield	UWDeepfield/_Activity/import-aad9f63e…	2,727	13 kB
   uwdeepfield	UWDeepfield/_Activity/import-af545232…	2,581	114 kB
   uwdeepfield	UWDeepfield/_Activity/import-e90bb2a9…	2,222	104 kB

   Q2 — do the writes grow? Yes, ~5× across the population

   writes per node	nodes	avg content	max content
   <10	24,478	8.5 kB	499 kB
   10–49	618	3.8 kB	641 kB
   50–99	122	16 kB	29 kB
   100–299	282	14 kB	110 kB
   300–599	80	83 kB	225 kB
   600+	33	52 kB	204 kB

   A rising curve, which per the framing settles it: this is a few writers emitting ever-larger serialisations, not an inherently slow path.

   Q3 — bytes attribution: the top 20 are 100% import activities

   path	writes	content	est. total serialised
   AgenticEngineering/_Activity/import-69f6248b…	5,239	141 kB	719 MB
   UWDeepfield/_Activity/import-af545232…	2,581	114 kB	288 MB
   UWDeepfield/_Activity/import-e90bb2a9…	2,222	104 kB	226 MB
   MeshWeaver/_Activity/import-1d1a65773fb0c6b6-…	637	204 kB	127 MB

   Not one non-import node in the top 20. Totals: 25,579 Activity nodes, 230,674 writes, ≈7.1 GB of JSON serialised (≈3.5 GB if content grew linearly from empty, which is the fairer integral). One activity node alone accounts for ~0.7 GB.

   The mechanism — O(n²), but NOT via patch size

   The patch-bloat hypothesis is falsified. PatchExtensions.DiffArray is element-wise: for an append it walks the equal prefix and emits a single add op, so the RFC 6902 output stays O(1) per append. The O(n²) is entirely on the input side of CreateSingleObjectPatch → ToJsonPatch:

   1. SerializeToNode the entire node content — O(n) bytes on every write
   2. diff walks all n existing elements — O(n) comparisons on every write
   3. → over n appends: O(n²) CPU and allocation, with O(n) patch bytes emitted

   That is exactly why a stack profile under-weights it and why #1292's ~20% per-call diff win did not move the needle: the dominant cost is re-serialising a growing document, not the diff algorithm. It also explains the ~1.4 cores of continuous GC — a JsonDocument/JsonObject graph per write, 230k writes.

   The named writer

   NodeTypeCompilationActivity.AppendLog (src/MeshWeaver.Graph/Configuration/NodeTypeCompilationActivity.cs:106) does one stream.Update per message, appending to ActivityLog.Messages. StaticRepoImporter calls it per item — per pruned node, per kept node, per failed import (StaticRepoImporter.cs:970, 1010, 1066, 1088, 1284, 1315). An import touching N nodes therefore performs N whole-document serialisations of a document that grows with N.

   The framework already says this is wrong. NodeTypeCompilationActivity.Complete (same file, line 153) exists precisely for it, and its own doc comment reads: "Use this instead of N separate AppendLog calls."

   The fix, and the one call I am not making

   The lever is fewer/larger-grained writes — per-call efficiency is already optimised and relocating to IIoPool does not help (and, as noted before, is a semantics change, since the patch must be posted in order).

   Two candidates:

   • (a) Batch the per-item lines. The prune loop already accumulates its paths, so the messages can be appended once per phase with Messages.AddRange — same content, same order, one write instead of N. Turns O(n²) into O(n). Cost: progress granularity drops from per-item to per-phase.
   • (b) Move log messages to satellite nodes (SatelliteEntityPatterns), making each append O(1) and preserving per-item live progress. Architecturally the right shape; a larger change to the ActivityLog contract, its readers, and the GUI.

   (a) is a small behaviour-visible trade (live per-item progress during long imports); (b) is a contract change. That is a scope call rather than a defect fix, so I am stopping here and naming it rather than picking one. Everything needed to choose is above.

   _{Method: aggregated over all *.activities tables via a temp table. The psql probe pod carried the DB credential in its env and has been deleted.}
7. rbuergi commented on Aug 12, 2026

   

   rbuergi

   on Aug 12, 2026

   Contributor

   More actions

   Correcting the record: the 64 is ours, it is a log-line latch, and nothing blocks at it

   This issue's body attributes the failure to "the Orleans NonReentrancyQueueSize limit" and to
   "a delivery leg is not completing". Both are wrong, and because they are stated with high
   confidence in an auto-filed issue they have already sent three separate investigations down the
   wrong path. Read from the code, here is what the signature actually is.

   1. 64 is RoutingGrain.SaturationThreshold — a MeshWeaver constant, not an Orleans limit

   src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs:85:

   internal const int SaturationThreshold = 64;

   Not configurable, and not Orleans' NonReentrancyQueueSize. The routing IIoPool is capped at
   256 (IoPoolOptions.Routing), and NodeType compilation runs on a separate pool
   (IoPoolNames.Compile, ProcessorCount) — the two share no lock and no pool.

   Nothing throttles, queues or refuses at 64. Dispatch hands every route to the pool
   unconditionally and RouteMessage still returns Forwarded immediately. The threshold gates a log
   line and nothing else. So "once the in-flight count hits 64, all further routing on that silo
   stalls" (issue body) does not happen.

   2. "Always exactly 64" is an artefact of the latch — this is the single most misleading datum here

   The issue treats "all 10 recorded samples show exactly 64" as the strongest evidence for a hard cap.
   It is the opposite: it is forced by the reporting code.

   if (inFlight < SaturationThreshold) return;                          // only ≥64 proceeds
   if (Interlocked.Exchange(ref saturationReported, 1) == 1) return;    // …and only ONCE
   logger.LogCritical(..., inFlight, ...);                              // prints the crossing value

   ReportSaturation is called with the post-increment value, fires on the single increment that
   crosses the line, and then latches. Under unit increments 64 is the only value it can ever
   print — the count could reach 500 and the line would still say 64. The number carries no
   information about depth at all.

   3. The prod evidence disproves the "wedge" reading

   ReportDrained only unlatches at half the threshold (inFlight > SaturationThreshold / 2
   returns early). So each additional critical requires the count to have fallen to ≤32 and climbed
   back to 64.

   Pod memex-portal-deployment-6bd9c46f8b-z9t6r logged five criticals between 11:45:09Z and
   11:45:27Z. That requires four separate drops below 32 in 18 seconds. Legs were terminating and
   draining en masse the entire time. Whatever this is, it is not a stuck delivery leg.

   4. The named address is the latest dispatch target — never a slow one

   The message interpolates the address of the dispatch that happened to cross the threshold. It is not
   selected for being slow, old, or stuck. The observation that the targets "span every module on the
   platform" is therefore expected under load and is not evidence about compilation — it just means
   those were the busiest addresses at the moment of crossing.

   Worth adding: _Activity/compile addresses take the grain branch, not the ordered stream branch
   (only portal, client, cache, mesh and import are stream-routed). That branch is bounded by
   ResolveTimeout (30 s) and its leg terminates as soon as path resolution emits — the grain hand-off
   is detached. So head-of-line blocking cannot explain the very targets this issue is named after.

   5. What the counter is actually sensitive to

   A slot is claimed at dispatch — and for the stream branch at enqueue, before the leg is
   subscribed at all. Meanwhile the leg's own bounds (ResolveTimeout, StreamPostTimeout) are Rx
   operators inside the cold observable, so they do not start until
   IIoPool.SubscribeThroughPool obtains a ThreadPool thread and passes the gate. That
   claim→subscribe window has no bound of any kind.

   So inFlightRoutes is partly an instrument for ThreadPool scheduling latency — the same quantity
   Orleans' LocalSiloHealthMonitor reports as a "thread pool delay". That is why this issue and #1284
   always appear together: they are two instruments reading one gauge, not cause and effect.

   The actual root cause, and where it stands

   @rbuergi's profiling comments above got this right: the silo is CPU-starved, not deadlocked, and
   the burner is O(node-size) patch construction on hub action-block turns — every stream.Update
   re-serialises the whole node to compute its patch, so a writer that appends to a growing collection
   is O(n²). The dominant instance (the static-repo importer's per-item activity log) was fixed in
   #1341.

   One real defect found on this path — a genuine slot leak

   IoPool.SubscribeThroughPool swallowed the drain's OperationCanceledException without
   terminating the observer. The drain registration only disposes the inner subscription, and
   disposal emits nothing — so the observable terminated neither completed nor errored, and every
   .Finally(...) hanging off it never ran. That callback is exactly what releases the in-flight route
   slot and advances OrderedRouteDispatcher's per-destination FIFO, so a leg the drain cancelled
   leaked its slot and stranded its destination's queue permanently.

   Fixed in #1358, together with rewriting this report so it states what it measures — in-flight
   count, stream destinations queued, deepest per-destination queue, pool occupancy, and (on clearing)
   how long the episode lasted. Deepest > 1 is the discriminator between "legs waiting on a leg" and
   "a busy silo"; the episode duration separates a two-second burst from a real stall without profiling
   a live pod. Repro: RoutingBackpressureShapeTest builds the two opposite shapes that yield an
   identical in-flight count of 64 and asserts the count cannot tell them apart while the snapshot can.

   What remains

   Only the rest of the O(n²) write class — same shape as #1341, tracked separately:
   ContentIndexingActivity.Append, GitSync ActivityRunner.Append,
   InstanceSyncService.AppendPending, the thread heartbeats, ActivityLogLogger, and
   ThreadExecution.PushToResponseMessage (which needs a delta field rather than batching).
   ActivityLog.Messages being an unbounded list inside node content is the general form of the
   problem.
8. rbuergi

   closed this as completedin #1358on Aug 13, 2026
9. rbuergi commented on Sep 19, 2026

   

   rbuergi

   on Sep 19, 2026

   Contributor

   More actions

   🚨 This ticket's occurrence count is wrong by six orders of magnitude, and has been since 2026-09-17.

   Read from the control instance's own record, Admin/_LogIncident/e4a97855ab595beb, 2026-09-19:

   field	value
   occurrences	13,108,032
   occurrencesAtLastComment	22
   lastSeen	2026-09-19T05:43:46Z
   error	Cannot comment: the incident has no repository.
   refusalAnnouncedAt	2026-09-17T19:09:17Z

   The incident node was created 2026-08-10, before LogIncident.Repository existed, so it carries the repository only inside its issueUrl — which points here. LogIncidentFiler.Comment refused to read it, and because an incident with an issue link is only ever offered a Comment and never a File, it could never recover: every recurrence re-entered the same method and re-issued the same refusal. The node has been rewritten roughly once a minute by that retry ever since (version 51438 and climbing).

   So the "22 occurrences" in the body above is the count as of the last comment that landed, not the count. Nothing on the ticket said so.

   Fixed in Systemorph/MeshWeaver.Plugins#2151 — the repository is read back out of the issue url and carried forward, so the node heals on the next recurrence and the retry ends because the comment lands.

   Two things for whoever works this issue:

   1. Do not read the absence of recent comments as the fault having stopped. It fired 13.1M times.
   2. Recurrence folding is currently stalled fleet-wide (Systemorph/MeshWeaver.Plugins#2152 — nothing has been folded since 07:28:51Z), so this ticket will not self-correct until that is resolved and Factor the default Blazor control views into MeshWeaver.Blazor.Views (view-pack module) #2151 ships.
10. systemorph-com commented on Sep 20, 2026

    

    systemorph-comBot

    on Sep 20, 2026

    Contributor

    More actions

    Reopened — this fault is happening again.

    Still occurring: 13108015 more since the last update (13108037 total).

    • Last seen: 2026-09-19 06:57:46Z
    • Pods: memex-portal-deployment-6c99c9f796-mh5cs, memex-portal-deployment-6956d54bb5-l6mxv, memex-portal-deployment-8476467cb6-nkbhh, memex-portal-deployment-6bd9c46f8b-z9t6r, memex-portal-deployment-67464cbcfc-jlmpw, memex-portal-deployment-59479dc7d9-kmwvr, memex-portal-deployment-59cf5fd6cf-gdl2s, memex-portal-deployment-748bfcc86b-2m6vg, memex-portal-deployment-c9c6b59c4-qpmks, memex-portal-deployment-686bdb74cb-vj9xl, memex-portal-deployment-76666bd6f5-mghv8, memex-portal-deployment-657845466c-dmvbk, memex-portal-deployment-78bb6bfd5-fpvxk, memex-portal-deployment-cc58dc7c6-ckll8, memex-portal-deployment-bdc9f8dc4-fdkgw, memex-portal-deployment-68584c4db4-mdmfz, memex-portal-deployment-69c694489b-nkkf5, memex-portal-deployment-7d4b5684bb-9lrxl, memex-portal-deployment-7574f9474c-zvqzn, memex-portal-deployment-585f6d7b98-z7ktk, memex-portal-deployment-7745468d76-m65r5, memex-portal-deployment-77c58c4469-g2ngb, memex-portal-deployment-797df55dfb-cn4p8, memex-portal-deployment-97c56b569-tjnmj, memex-portal-deployment-856988c494-tfsqs, memex-portal-deployment-856988c494-sr9pt, memex-portal-deployment-7799dd87b9-nstzw, memex-portal-deployment-f858c9f46-7jtfp, memex-portal-deployment-6db49f988f-44qkf, memex-portal-deployment-74c7d69bf5-fjvzh, memex-portal-deployment-84779c8448-dhrv5, memex-portal-deployment-5d6f77b68c-srtqg, memex-portal-deployment-696cc645c6-gfw6n, memex-portal-deployment-c89ddfbb6-vzqm7, memex-portal-deployment-f47f8cdd-f8fmq, memex-portal-deployment-89d7df89f-lrbc4, memex-portal-deployment-89d7df89f-k9fkr, memex-portal-deployment-76b967b5fb-pkw5p, memex-portal-deployment-76b967b5fb-h8kzg, memex-portal-deployment-745df44fc8-r4w7t, memex-portal-deployment-745df44fc8-drjtp, memex-portal-deployment-745df44fc8-gtrfd, memex-portal-deployment-5b955b6d44-dzxsl, memex-portal-deployment-98785f547-76qjj, memex-portal-deployment-ddbc747c4-2thwc, memex-portal-deployment-868d46474d-w8dc2, memex-portal-deployment-56985b9df8-q4dcr, memex-portal-deployment-7bcb749b9f-7fqd5, memex-portal-deployment-7bcb749b9f-nch9z, memex-portal-deployment-775f5487ff-2mdwj, memex-portal-deployment-59ccff47c5-rjs77, memex-portal-deployment-59ccff47c5-xw4zx, memex-portal-deployment-59ccff47c5-k8bz8, memex-portal-deployment-59ccff47c5-4dh52, memex-portal-deployment-7db656d9d9-nbfzk, memex-portal-deployment-5745957455-2xwk4, memex-portal-deployment-565b67998c-24vwd, memex-portal-deployment-d7dd9796b-66g44, memex-portal-deployment-d7dd9796b-rc4bh, memex-portal-deployment-565b67998c-rbsd7, memex-portal-deployment-565b67998c-zzjl4, memex-portal-deployment-866fdb8985-lq6pz, memex-portal-deployment-565b67998c-2kvsk, memex-portal-deployment-5d6dbfc4c8-5vllq, memex-portal-deployment-65fbcf6cd5-spgc9, memex-portal-deployment-55f6c4c4c5-wcccn, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-7d6b864b54-gr6n6, memex-portal-deployment-7897bfd494-x8rfw, memex-portal-deployment-7897bfd494-dl9wb, memex-portal-deployment-688b496b7-2sdt8, memex-portal-deployment-7d6b864b54-w4xnl, memex-portal-deployment-688b496b7-brp4j, memex-portal-deployment-999c9c79d-8ccj2, memex-portal-deployment-999c9c79d-g9wps, memex-portal-deployment-6dc7cf965d-zwv4x, memex-portal-deployment-64d9857f9d-v7hmx, memex-portal-deployment-54cc7585b4-2mbtc, memex-portal-deployment-df7ddfdf4-xb9bk, memex-portal-deployment-df7ddfdf4-dtp44, memex-portal-deployment-df7ddfdf4-srw4s, memex-portal-deployment-df7ddfdf4-xtjhc, memex-portal-deployment-686bf68bdc-8jj6b, memex-portal-deployment-574b6d7fdf-nqbg4, memex-portal-deployment-686bf68bdc-4d9s2, memex-portal-deployment-6d58c699f8-bp5bs, memex-portal-deployment-6f55d94c6c-cbprp, memex-portal-deployment-775748f44d-jn76d, memex-portal-deployment-6d58c699f8-9t675, memex-portal-deployment-775748f44d-7xlkm, memex-portal-deployment-696766b96f-6s2xs, memex-portal-deployment-696766b96f-mv8z4, memex-portal-deployment-7cfbf545d9-gtlhr, memex-portal-deployment-76cbbdd5f-82mvq, memex-portal-deployment-76cbbdd5f-9mtkj, memex-portal-deployment-569544576b-q6nvd, memex-portal-deployment-569544576b-9h52b, memex-portal-deployment-569544576b-89r5k, memex-portal-deployment-7cfbf545d9-jk5qc, memex-portal-deployment-867dc45877-lgskl, memex-portal-deployment-867dc45877-hz7wt, memex-portal-deployment-867dc45877-fmcss, memex-portal-deployment-5b6c4f89f8-hpcbf, memex-portal-deployment-58c7bb4b9d-97pjf, memex-portal-deployment-58c7bb4b9d-z2xcv, memex-portal-deployment-5b6c4f89f8-rpfmq, memex-portal-deployment-6f64bf4b56-gxmrr, memex-portal-deployment-6f64bf4b56-2mzrf, memex-portal-deployment-6f64bf4b56-xqw96, memex-portal-deployment-6b8cfb8bfc-bnlzs, memex-portal-deployment-858f57dc6-8zlfh, memex-portal-deployment-6b8cfb8bfc-ttw4m, memex-portal-deployment-858f57dc6-wns9j, memex-portal-deployment-7b67cb5577-cqzp2, memex-portal-deployment-7544fdd47b-bxtjf, memex-portal-deployment-858f57dc6-6wggn, memex-portal-deployment-858f57dc6-ff5kn, memex-portal-deployment-6dc5679f77-nd67w, memex-portal-deployment-858f57dc6-bkkfv, memex-portal-deployment-6fd666bc95-f7brs, memex-portal-deployment-6fd666bc95-s79r4, memex-portal-deployment-56d4694745-ss8mq, memex-portal-deployment-79b5fd5f6b-jg4td, memex-portal-deployment-79b5fd5f6b-7wdn4, memex-portal-deployment-7b5f547cb7-cp2wv, memex-portal-deployment-6dc5679f77-vf2qx, memex-portal-deployment-6c7b9889c7-72fmv, memex-portal-deployment-cf5dc996f-gk2bv, memex-portal-deployment-6c46b84848-hnh9v, memex-portal-deployment-6c46b84848-c9mm5, memex-portal-deployment-6bfbc8ff4b-rz9p4, memex-portal-deployment-6bfbc8ff4b-vvc82, memex-portal-deployment-86779d495b-ksxqq, memex-portal-deployment-595d8b9995-j5f9p, memex-portal-deployment-86779d495b-qrnh6, memex-portal-deployment-7d95465fbb-qrhvg, memex-portal-deployment-7d95465fbb-s7gds, memex-portal-deployment-5df4ccb89-z8w2k, memex-portal-deployment-5df4ccb89-sprw2, memex-portal-deployment-76c65fb457-jr82z, memex-portal-deployment-76c65fb457-2w858, memex-portal-deployment-76c65fb457-z8cw8, memex-portal-deployment-558d86bb48-vl44h, memex-portal-deployment-558d86bb48-nxpcc, memex-portal-deployment-788cffc666-xbmfs, memex-portal-deployment-788cffc666-swfdl, memex-portal-deployment-788cffc666-46jqr, memex-portal-deployment-5d4bfc98b7-5r56d, memex-portal-deployment-5d4bfc98b7-ff447, memex-portal-deployment-788cffc666-kh7sl, memex-portal-deployment-788cffc666-9zhzw, memex-portal-deployment-64cc8b9b6d-z9w44, memex-portal-deployment-64cc8b9b6d-rk5gw, memex-portal-deployment-64cc8b9b6d-q264s, memex-portal-deployment-54cc8d6b4d-qsxvp, memex-portal-deployment-6c954fd5bf-7b4xq, memex-portal-deployment-6c954fd5bf-b5q52, memex-portal-deployment-565c5d6b97-26bkz, memex-portal-deployment-565c5d6b97-zwvjs, memex-portal-deployment-54cc8d6b4d-cjnqn, memex-portal-deployment-565c5d6b97-wp8g7, memex-portal-deployment-7f6d66995c-dbzj8, memex-portal-deployment-7f6d66995c-r8zmq, memex-portal-deployment-7f6d66995c-tvd9l, memex-portal-deployment-d999f9888-n6ggc, memex-portal-deployment-d999f9888-f86ts, memex-portal-deployment-d999f9888-sfpph, memex-portal-deployment-85bd9f5d85-6th26, memex-portal-deployment-59ddc494b7-r6gqg, memex-portal-deployment-5998589dbd-tzx4w, memex-portal-deployment-5998589dbd-9v8mj, memex-portal-deployment-5678bbc9c8-pl7dq, memex-portal-deployment-dbc8997cd-q2n4s, memex-portal-deployment-6c74bf4c66-88jg8, memex-portal-deployment-84765b45c8-f4j9m, memex-portal-deployment-fbfc4b45-mqzbp, memex-portal-deployment-856d4474c-hr5rd, memex-portal-deployment-dbc8997cd-w55hb, memex-portal-deployment-7858d9f8cc-r2v5v, memex-portal-deployment-7858d9f8cc-h6gp2, memex-portal-deployment-7d8fd6f84b-mjp87, memex-portal-deployment-7d8fd6f84b-47cmq, memex-portal-deployment-856d4474c-4rzkz, memex-portal-deployment-7657b88b9-xqt9w, memex-portal-deployment-7657b88b9-qwcxz, memex-portal-deployment-69f86967d8-9xxb7, memex-portal-deployment-69f86967d8-vmj29, memex-portal-deployment-69f86967d8-54p5q, memex-portal-deployment-6f76b9c79b-9spms, memex-portal-deployment-865476c896-6bv8p, memex-portal-deployment-865476c896-vxsr4, memex-portal-deployment-865476c896-xbbsw, memex-portal-deployment-5f9479f848-hmmbn, memex-portal-deployment-5f9479f848-fsvjr, memex-portal-deployment-7ccdd994cf-sbrjg, memex-portal-deployment-7ccdd994cf-wwwsj, memex-portal-deployment-768b8dff99-h48bj, memex-portal-deployment-7c6b48957d-xqzdq, memex-portal-deployment-7c6b48957d-c58hl, memex-portal-deployment-7c6b48957d-n5vkr, memex-portal-deployment-768b8dff99-cn8nn, memex-portal-deployment-7c6b48957d-9zc78, memex-portal-deployment-7c6b48957d-rpkhr, memex-portal-deployment-78cd7b4f8f-g85gw, memex-portal-deployment-767d8d6684-gdl2g, memex-portal-deployment-767d8d6684-s6kcf, memex-portal-deployment-78cd7b4f8f-npfqs, memex-portal-deployment-6d55467d9d-46jfk, memex-portal-deployment-767d8d6684-vrc78, memex-portal-deployment-c4499cbcc-fq9xr, memex-portal-deployment-5d99f5649-t5h8n, memex-portal-deployment-cfc9f8448-d4txb, memex-portal-deployment-67c44ccd96-z9g2b, memex-portal-deployment-cfc9f8448-vjlb9, memex-portal-deployment-554545c8cc-xw7xn, memex-portal-deployment-554545c8cc-m7kw2, memex-portal-deployment-56699545bf-pk948, memex-portal-deployment-76c67b6b45-5n2zg, memex-portal-deployment-66cfbf759-svxh9, memex-portal-deployment-76c67b6b45-v4hdf, memex-portal-deployment-66cfbf759-cnwnb, memex-portal-deployment-66f4b5c9b6-4k6d5, memex-portal-deployment-66f4b5c9b6-mcm2z, memex-portal-deployment-66cfbf759-gmgl4, memex-portal-deployment-658bbfd875-xpf9w, memex-portal-deployment-799c674567-bszk6, memex-portal-deployment-79776f9c5f-vxtfc, memex-portal-deployment-79776f9c5f-ztq9j, memex-portal-deployment-799c674567-v8l2n, memex-portal-deployment-7767bf7cd7-nx4rw, memex-portal-deployment-8f976f958-j28fg, memex-portal-deployment-8f976f958-p4vvt, memex-portal-deployment-b74d87c5c-n8vqj, memex-portal-deployment-b74d87c5c-5qjxx, memex-portal-deployment-7767bf7cd7-rwhxm, memex-portal-deployment-75597864c8-66ddg, memex-portal-deployment-6dd4fbd566-wv9p7, memex-portal-deployment-6dd4fbd566-gqcsm, memex-portal-deployment-9bf79b774-tk88b, memex-portal-deployment-9bf79b774-nvnzz, memex-portal-deployment-75d46686fb-5gpz8, memex-portal-deployment-644ffc96b9-58pv4, memex-portal-deployment-9bf79b774-jwrng, memex-portal-deployment-9bf79b774-s9sr9, memex-portal-deployment-644ffc96b9-mttj7, memex-portal-deployment-9bf79b774-b8dl6, memex-portal-deployment-9bf79b774-bwgqp, memex-portal-deployment-67dbc7bc9c-455hx, memex-portal-deployment-67dbc7bc9c-wcg8k, memex-portal-deployment-85859db964-7sqbk, memex-portal-deployment-55547495d7-c8z9r, memex-portal-deployment-85859db964-q5zcm, memex-portal-deployment-55547495d7-v2qqt, memex-portal-deployment-85859db964-mgxj7, memex-portal-deployment-85859db964-rfjgc, memex-portal-deployment-c7f5ff965-vk9zq, memex-portal-deployment-85859db964-qmvw5, memex-portal-deployment-c7f5ff965-7kj5l, memex-portal-deployment-7b964c8947-pl825, memex-portal-deployment-578b7bc84b-msvh5, memex-portal-deployment-58c88885f8-h5hfs, memex-portal-deployment-85859db964-rfsd7, memex-portal-deployment-58c88885f8-lhgp7, memex-portal-deployment-85859db964-xk5gw, memex-portal-deployment-67b654495f-d8g8n, memex-portal-deployment-b964b4c4c-wmrhr, memex-portal-deployment-69cb4d6486-lj2sb, memex-portal-deployment-7c9696d89b-zskgh, memex-portal-deployment-77cfc55bfc-5ztnt, memex-portal-deployment-77cfc55bfc-txpk7, memex-portal-deployment-5c8c476d54-h22p6, memex-portal-deployment-77cfc55bfc-767mq, memex-portal-deployment-77cfc55bfc-92tsr, memex-portal-deployment-5c8c476d54-55h26, memex-portal-deployment-77898c4947-bfnft, memex-portal-deployment-5b9cbf46dd-5nqbz, memex-portal-deployment-5b9cbf46dd-gs6j6, memex-portal-deployment-5b9cbf46dd-q8v6s, memex-portal-deployment-5b9cbf46dd-fxbhd, memex-portal-deployment-5b9cbf46dd-zn7nf, memex-portal-deployment-77898c4947-6mffn, memex-portal-deployment-5b9cbf46dd-xrgsx, memex-portal-deployment-5fc89b764-dcdcq, memex-portal-deployment-5fc89b764-sb7cz, memex-portal-deployment-58c859d4cc-b8pfg, memex-portal-deployment-58c859d4cc-wgcwc, memex-portal-deployment-58c859d4cc-l2rmp, memex-portal-deployment-58c859d4cc-hlv2q, memex-portal-deployment-58c859d4cc-xsvkj, memex-portal-deployment-58c859d4cc-gqqg7, memex-portal-deployment-58c859d4cc-bml8l, memex-portal-deployment-58c859d4cc-tqhzx, memex-portal-deployment-58c859d4cc-sb9qb, memex-portal-deployment-79ddd44d77-xzp5d, memex-portal-deployment-79ddd44d77-k8qlc, memex-portal-deployment-58c859d4cc-w7m5t, memex-portal-deployment-58c859d4cc-5vfjp, memex-portal-deployment-f69cd9d56-hg9bc, memex-portal-deployment-79ddd44d77-qwdjd, memex-portal-deployment-6d96874b7f-ldw5f, memex-portal-deployment-7954bc594c-8dx4g, memex-portal-deployment-5f455dbc49-bzt6n, memex-portal-deployment-5f455dbc49-pc2cm, memex-portal-deployment-7bff7d8b54-p5hkc, memex-portal-deployment-64864758dc-q2kn5, memex-portal-deployment-6457bb9544-g6z76, memex-portal-deployment-6457bb9544-qpdjd, memex-portal-deployment-6457bb9544-8plns, memex-portal-deployment-6457bb9544-w4wnn, memex-portal-deployment-6457bb9544-hnqcf, memex-portal-deployment-7bff7d8b54-5rrkx, memex-portal-deployment-84d966d9bd-nlv6z, memex-portal-deployment-74bfcd8f6b-sksm4, memex-portal-deployment-74bfcd8f6b-gkz48, memex-portal-deployment-74bfcd8f6b-95bc8, memex-portal-deployment-74bfcd8f6b-dftr8, memex-portal-deployment-776b97b9fb-k98fs, memex-portal-deployment-d6bfd5dcb-glbvw, memex-portal-deployment-d6bfd5dcb-n7ptw, memex-portal-deployment-d6bfd5dcb-qk5gf, memex-portal-deployment-d6bfd5dcb-x4hxp, memex-portal-deployment-75f7864cfd-n5ws6, memex-portal-deployment-75f7864cfd-pcxnd, memex-portal-deployment-8b96fdbd4-h7cxj, memex-portal-deployment-d6bfd5dcb-vshjn, memex-portal-deployment-7d5d458cc4-cbztk, memex-portal-deployment-d6bfd5dcb-m7cj5, memex-portal-deployment-6c74f8458-96jq9, memex-portal-deployment-7fd85c4468-gwghs, memex-portal-deployment-59955df69b-vllwd, memex-portal-deployment-74c97ccd44-6fvqj, memex-portal-deployment-59955df69b-t2pbb, memex-portal-deployment-74c97ccd44-nmjks, memex-portal-deployment-5777bff985-88g9r, memex-portal-deployment-5777bff985-n98c5, memex-portal-deployment-bf77c847-xsb54, memex-portal-deployment-bf77c847-tctvp, memex-portal-deployment-bf77c847-zw22p, memex-portal-deployment-76c99bdc6-9stvn, memex-portal-deployment-76c99bdc6-g7f89, memex-portal-deployment-bf77c847-6zx7c, memex-portal-deployment-bf77c847-4sd7q, memex-portal-deployment-bf77c847-f286h, memex-portal-deployment-bf77c847-m5l4r, memex-portal-deployment-6cdf7f9f-8dztq, memex-portal-deployment-bf77c847-gh7g6, memex-portal-deployment-6cdf7f9f-2g8bb, memex-portal-deployment-bf77c847-bq5tr, memex-portal-deployment-5f49fd47c4-g4ct8, memex-portal-deployment-5f49fd47c4-jnpm4, memex-portal-deployment-6d4564c7b8-cvdqh, memex-portal-deployment-bf77c847-8g4vz, memex-portal-deployment-6d4564c7b8-9dv6t, memex-portal-deployment-6546ff4bc8-cj78b, memex-portal-deployment-6546ff4bc8-9wcwl, memex-portal-deployment-5bf869698f-5qgvz, memex-portal-deployment-5bf869698f-6x87d, memex-portal-deployment-bf77c847-r5gcw, memex-portal-deployment-59d8fd958f-sszkb, memex-portal-deployment-59d8fd958f-b2z6v, memex-portal-deployment-7456f8f447-26bbj, memex-portal-deployment-7456f8f447-6pkwq, memex-portal-deployment-7456f8f447-ctqhw, memex-portal-deployment-7456f8f447-pjb9h, memex-portal-deployment-7456f8f447-bs6p6, memex-portal-deployment-7456f8f447-lgtrr, memex-portal-deployment-685fb69888-sjl6z, memex-portal-deployment-685fb69888-r8wl9, memex-portal-deployment-685fb69888-f8xxd, memex-portal-deployment-84d46fdb95-f9lb5, memex-portal-deployment-84d46fdb95-mvf8b, memex-portal-deployment-84d46fdb95-sd7p4, memex-portal-deployment-7ffbd9b949-nqkvb, memex-portal-deployment-7ffbd9b949-rvqk8, memex-portal-deployment-5867566569-k9nmf, memex-portal-deployment-5867566569-frfqx, memex-portal-deployment-5867566569-c9xbt, memex-portal-deployment-7f74766b8d-rmjfc, memex-portal-deployment-7f74766b8d-rltnt, memex-portal-deployment-67f7fc74f6-cqwr5, memex-portal-deployment-67f7fc74f6-cvvnq, memex-portal-deployment-574d54c98d-cdd47, memex-portal-deployment-574d54c98d-vvcwq, memex-portal-deployment-6d7497cb58-jc296, memex-portal-deployment-747c879d8c-ph572, memex-portal-deployment-747c879d8c-w2vsm, memex-portal-deployment-d8bf579fc-8lpl4, memex-portal-deployment-77858cc9cd-9p2xt, memex-portal-deployment-7dbb4cf6b4-ndhjx, memex-portal-deployment-584db8fc55-db7jd, memex-portal-deployment-584db8fc55-m2g2s, memex-portal-deployment-584db8fc55-vtslp, memex-portal-deployment-868794b948-lnwmb, memex-portal-deployment-868794b948-mrrkd, memex-portal-deployment-849544cd4-wlwp5, memex-portal-deployment-75dff475b6-dg5r9, memex-portal-deployment-75dff475b6-lckkp, memex-portal-deployment-75dff475b6-glxdr, memex-portal-deployment-8ff67f8c7-vm8d7, memex-portal-deployment-8ff67f8c7-gcf24, memex-portal-deployment-5f5fdfbd98-9vkkc, memex-portal-deployment-5f5fdfbd98-ghh75, memex-portal-deployment-bfc5dcf6f-7z88q, memex-portal-deployment-7fd9668b69-jwzzl, memex-portal-deployment-7fd9668b69-sjm5v, memex-portal-deployment-7fd9668b69-jnwlv, memex-portal-deployment-6c7d6b6b44-klrt7, memex-portal-deployment-7fd9668b69-lzm7l, memex-portal-deployment-7fd9668b69-78tr4, memex-portal-deployment-6c7d6b6b44-c47v8, memex-portal-deployment-7fd9668b69-j4fdv, memex-portal-deployment-7fd9668b69-4qtkx, memex-portal-deployment-7fd9668b69-v72dx, memex-portal-deployment-854cd8cd6d-r4cn5, memex-portal-deployment-854cd8cd6d-8bc2q, memex-portal-deployment-6546c779d4-4ngqv, memex-portal-deployment-6546c779d4-2dd5f, memex-portal-deployment-9c66fbf57-7c7tl, memex-portal-deployment-9c66fbf57-9gcfq, memex-portal-deployment-5d786cfbdf-cm8b6, memex-portal-deployment-8468dffdb4-cgzsc, memex-portal-deployment-5d786cfbdf-rv5d7, memex-portal-deployment-8468dffdb4-fk9x5, memex-portal-deployment-8468dffdb4-sp5vf, memex-portal-deployment-8468dffdb4-f9hdm, memex-portal-deployment-7fd9668b69-gwwmq, memex-portal-deployment-7fd9668b69-dtnb6, memex-portal-deployment-7fd9668b69-xx2qv, memex-portal-deployment-7fd9668b69-mc7dr, memex-portal-deployment-7fd9668b69-5qd97, memex-portal-deployment-7cb6684584-jdw7v, memex-portal-deployment-7cb6684584-ztqz8, memex-portal-deployment-7fd9668b69-t7brm, memex-portal-deployment-7fd9668b69-2xngb, memex-portal-deployment-7fd9668b69-9ztfc, memex-portal-deployment-7fd9668b69-kvwgv, memex-portal-deployment-6c967c7c4c-kdff5, memex-portal-deployment-6c967c7c4c-2k4xb, memex-portal-deployment-54787585bc-qfwxx, memex-portal-deployment-54787585bc-m64r2, memex-portal-deployment-54787585bc-gh8pr, memex-portal-deployment-ddd9f5f75-tfxpq, memex-portal-deployment-6b879996c7-dstvb, memex-portal-deployment-d77fd7c88-x7nzm, memex-portal-deployment-6b879996c7-cmnfg, memex-portal-deployment-6b879996c7-lk6fd, memex-portal-deployment-565f77b9c9-vfrfn, memex-portal-deployment-bbf878dfb-gknl9, memex-portal-deployment-bbf878dfb-ztq58, memex-portal-deployment-6b879996c7-scj5g, memex-portal-deployment-6b879996c7-bntfq, memex-portal-deployment-6b68cf8588-tqjxt, memex-portal-deployment-6b68cf8588-bgrjp, memex-portal-deployment-6b68cf8588-xb6wr, memex-portal-deployment-fb6488b5f-zzxjk, memex-portal-deployment-fb6488b5f-x7bv5, memex-portal-deployment-6b68cf8588-rqt25, memex-portal-deployment-6b68cf8588-zzs46, memex-portal-deployment-6b68cf8588-tdz4g, memex-portal-deployment-5fbfd78c98-xk69t, memex-portal-deployment-5fbfd78c98-dwhzb, memex-portal-deployment-6c99c9f796-6ldfk, memex-portal-deployment-5fbfd78c98-sc5fw, memex-portal-deployment-5fbfd78c98-8m4jz

    Recent log lines

    2026-09-19 05:55:46Z memex-portal-deployment-6c99c9f796-mh5cs crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
          [ROUTE] Routing back-pressure [98bcaac6#4 started 2026-09-19T05:55:46.6542832Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target rbuergi/_Install/DefaultViews — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
    2026-09-19 05:56:46Z memex-portal-deployment-6c99c9f796-mh5cs crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
          [ROUTE] Routing back-pressure [98bcaac6#5 started 2026-09-19T05:56:46.6958449Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Anthropic/_Entitlements/_Policy — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
    2026-09-19 06:57:46Z memex-portal-deployment-6c99c9f796-mh5cs crit: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
          [ROUTE] Routing back-pressure [98bcaac6#6 started 2026-09-19T06:57:46.6731617Z]: 64 route dispatches in flight (reporting threshold 64); stream destinations queued 0, deepest per-destination queue 0, routing pool subscribing 0. Latest dispatch target Edu/Page — the address that happened to cross the threshold, NOT a diagnosis. A slot is held from dispatch until the leg terminates, INCLUDING the unbounded wait for a ThreadPool thread before the leg's own timeouts start, so a CPU-starved silo raises this with nothing stuck. A deepest queue of 1 or more means legs are blocked behind a leg (head-of-line on one destination); 0 means nothing is waiting on anything, so read it as load. 🚨 Deepest is sampled AT THE CROSSING, so like the in-flight count it is partly an artefact of the threshold: with N destinations sharing the backlog it is ~InFlight/N whatever is wrong. READ THE EPISODE STAMP, not the depth: a later line with a HIGHER episode on this activation means this episode drained; a line with a DIFFERENT activation id means the grain was recycled; and if neither a clear nor a higher episode ever follows, the in-flight count never fell below half the threshold — which means a leg never terminated and its slot leaked, not that the silo was busy.
    

    _{Re-addressed by the current identity function: this incident inherited e4a97855ab595beb (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}
11. systemorph-com

    reopened this on Sep 20, 2026
12. rbuergi

    added

    bugSomething isn't workingSomething isn't working

    area:orleansCore subsystem: the Orleans silo - grains, placement, routingCore subsystem: the Orleans silo - grains, placement, routing

    feature:routing-back-pressureFeature: routing back pressureFeature: routing back pressure

    sev:HBLOCKING: primary path broken, intermittent user-visible failure, or silently wrong results.BLOCKING: primary path broken, intermittent user-visible failure, or silently wrong results.

    on Sep 20, 2026
13. rbuergi

    closed this as completedin #4973on Sep 21, 2026