Recurrence: Orleans silo thread-pool stall (~25s) in memex-portal — continued from #4886, risk of missed heartbeats and cluster eviction

What is failing

The Orleans silo in the memex-portal deployment hit a full .NET Thread Pool execution stall of ~24.7s, detected by Orleans.Runtime.MembershipService.LocalSiloHealthMonitor. During such a stall the silo runs nothing — including timer callbacks — risking missed membership heartbeats and cluster eviction.

Probable cause

Cause unclear from the log alone (the monitor reports the symptom, not the trigger). Plausible candidates: a very long Gen2/LOH GC pause, thread-pool starvation with slow ramp-up under a tight container CPU limit, or CPU-quota cgroup throttling. This occurrence is notably longer (~24.7s) than the previously filed cluster of stalls (11.5–24.9s), suggesting the condition is not improving.

Impact

Low frequency — a single new occurrence on one pod, but it continues a pattern that has now hit six different memex-portal pods since 2026-09-15. No observed outage yet, but a stall of this length in a silo hosting grains causes transient request timeouts and, if it exceeds the membership death window, cluster eviction and grain migration churn.

Duplicate / related

This is a recurrence of the fingerprint already filed as #4886 (incident Admin/_LogIncident/1dbb05f503932ad3, filed 2026-09-20, stalls on five pods). This ticket tracks the newest occurrence; a human may prefer to fold it into #4886 rather than track separately.

Where to look

Orleans.Runtime.MembershipService.LocalSiloHealthMonitor (Orleans silo health watchdog) — the detection/logging site.
Deployment CPU/memory limits of the memex-portal pods — check CPU throttling (container_cpu_cfs_throttled_periods_total) and GC stats at the stall timestamp.

Evidence


Fingerprint	`eda900bec4354ee0`
Category	`Orleans.Runtime.MembershipService.LocalSiloHealthMonitor`
Severity	Error
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-857546649-2c25b`
Occurrences	1
First seen	2026-09-22 18:47:10Z
Last seen	2026-09-22 18:47:10Z
Routing	not determined — no configured route matches the category `Orleans.Runtime.MembershipService.LocalSiloHealthMonitor`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-09-22 18:47:10Z memex-portal-deployment-857546649-2c25b fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
      .NET Thread Pool execution stalled for 24.6915577s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.

_{Re-addressed by the current identity function: this incident inherited 3a7d53ee7da8a1b4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

_{Opened automatically from Admin/_LogIncident/eda900bec4354ee0. Recurrences are folded into this issue rather than opening new ones.}

Triage 2026-09-23: same site and discriminator as #4886. Its one occurrence predates the fix candidate, so only the post-roll read remains

Same root as #4886. Both have the same log site, Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487] (".NET Thread Pool execution stalled for …"), the same namespace (memex-cloud) and the same measured quantity. This ticket's body proposes folding it into #4886. It stays open for now only because its remaining step is verification rather than code.

The occurrence predates the candidate fix on this namespace:


stall	2026-09-22 18:47:10Z, pod `memex-portal-deployment-857546649-2c25b`, 24.69 s
#5327 (NodeType compiles kept off the silo's shared ThreadPool)	merged 20:27:32Z as `ec191c2b61`, 1 h 40 min after the stall, so that pod cannot have carried it
memex.meshweaver.cloud `/api/version`	`39f49494eb`. `git merge-base --is-ancestor ec191c2b61 39f49494eb` holds, so it carries #5327
memex.systemorph.com `/api/version`	`3da3423c4b`. It does not contain `ec191c2b61`, so readings from `memex` say nothing about #5327
#5182 (`DrainLoop` bounded at 64 turns, `94689968ef`)	contained in both images

The read that closes or keeps this. It goes on the control instance, which answered 503 when I tried, so I did not take it:

{ "requestedAction": "Sample" } on Deployments/memex-cloud. Confirm that every replica runs 39f49494eb or later, and note each pod's startedAt.
{ "requestedAction": "Logs", "deployment": "Deployments/memex-cloud", "query": "Thread Pool execution stalled", "sinceMinutes": <minutes since the roll> }. Read the coverage the result reports (earliest row, cut). A sinceMinutes value is a request, not coverage.

How to read it.

Zero stall lines from pods started after the roll, over at least a day of real coverage, supports compile starvation as the trigger.
A stall on a 39f49494eb pod keeps Orleans silo thread-pool stalls (12–18s) across all memex-portal pods risk missed heartbeats and cluster eviction #4886 open and excludes that candidate.
fix(compiler): keep NodeType compiles off the silo's shared ThreadPool #5327 cannot help a stall caused by a GC pause. Orleans silo thread-pool stalls (12–18s) across all memex-portal pods risk missed heartbeats and cluster eviction #4886's occurrence 2 was a GC pause, and RoutingGrain route dispatch fails with OutOfMemoryException during pod-wide heap exhaustion — downstream of ObjectPolymorphicConverter OOM burst #4824 is its OOM correlate.

Not established: which of the three candidates (GC pause, pool starvation, CFS throttling) produced this 24.7 s stall. No GC, CPU-throttle or ThreadPool reading exists for 18:47Z.

Still occurring: 1 more since the last update (2 total).

Last seen: 2026-09-23 12:30:53Z
Pods: memex-portal-deployment-857546649-2c25b, memex-portal-deployment-6bf5d7dcc8-jl5k8

Recent log lines

2026-09-22 18:47:10Z memex-portal-deployment-857546649-2c25b fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
      .NET Thread Pool execution stalled for 24.6915577s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.
2026-09-23 12:30:53Z memex-portal-deployment-6bf5d7dcc8-jl5k8 fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
      .NET Thread Pool execution stalled for 15.0738807s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.

_{Re-addressed by the current identity function: this incident inherited 3a7d53ee7da8a1b4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Still occurring: 4 more since the last update (6 total).

Last seen: 2026-09-23 14:51:41Z
Pods: memex-portal-deployment-857546649-2c25b, memex-portal-deployment-6bf5d7dcc8-jl5k8, memex-portal-deployment-79889596-47trv, memex-portal-deployment-777677694-p6hbn

Recent log lines

2026-09-23 12:32:10Z memex-portal-deployment-6bf5d7dcc8-jl5k8 fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
      .NET Thread Pool execution stalled for 10.1493175s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.
2026-09-23 13:13:23Z memex-portal-deployment-79889596-47trv fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
      .NET Thread Pool execution stalled for 11.3939403s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.
2026-09-23 14:51:41Z memex-portal-deployment-777677694-p6hbn fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
      .NET Thread Pool execution stalled for 10.1619648s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.

_{Re-addressed by the current identity function: this incident inherited 3a7d53ee7da8a1b4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Canonical for the `LocalSiloHealthMonitor` stall site — #4886 folded here

#4886 (same site, Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487], memex-cloud, 3 stalls 2026-09-15 → 09-19) is closed as a duplicate of this ticket, because this is the issue the live incident node Admin/_LogIncident/eda900bec4354ee0 folds recurrences onto. Its analysis still applies and is not repeated here: occurrence 2 (12.6 s, 58dbb6d98d-8msww, 2026-09-19 06:29:21Z) was a full blocking GC inside #4824's OOM window (comment); the gauge is a symptom, not a trigger; grouping on #2480.

Reading the recurrences above against the fix candidates (measured now):

portal	`/api/version`	carries #5327 (`ec191c2b61`, compiles off the shared pool)?
memex.meshweaver.cloud (namespace memex-cloud)	`9a61fed964` (merged 2026-09-23 10:02Z)	yes (`git merge-base --is-ancestor`)
memex.systemorph.com (namespace memex)	`39f49494eb`	yes

The four 2026-09-23 stalls (12:30:53Z 15.1 s 6bf5d7dcc8-jl5k8, 12:32:10Z 10.1 s same pod, 13:13:23Z 11.4 s 79889596-47trv, 14:51:41Z 10.2 s 777677694-p6hbn) fall after 9a61fed964 existed, but I did not establish which image each of those pods ran (memex-cloud went through several ReplicaSets that day, and the systemorph control instance's Sample/Logs were not reachable from this session). If any of them ran an image containing ec191c2b61, compile starvation is excluded as the (sole) trigger and the remaining candidates are GC pause (cf. #4824 / #5555) and CFS throttling.

The read that decides it (owner / control instance): { "requestedAction": "Sample" } on Deployments/memex-cloud for pod → image, then { "requestedAction": "Logs", "query": "Thread Pool execution stalled", "sinceMinutes": … } and match each stall's pod to its image.

Post-roll reading 2026-09-24 ~06:51Z (memex-cloud): the Error-level stall is quiet, the starvation it gauges is not. Stays open

Image. Every memex-cloud pod that booted in the window ran core 9a61fed964, per the [PlatformStartup] version= lines (Ops/logs-memexcloud-20260924-verify-platformstartup, 38 boots). ReplicaSet 786d86cf54 is ci.9260, 6688fb9488 and 7d7f4d84f9 are ci.9262, and 6f497dc9b and 76cf776847 are ci.9263. 9a61fed964 contains #5327 (ec191c2b61).

Readings, 900 min (2026-09-23 ~15:45Z to 2026-09-24 06:51Z):

Ops/logs-memexcloud-20260924-verify-5388-10s, |~ "(?i)Thread Pool execution stalled for [1-9][0-9]+\.", which is this ticket's ≥10 s Error-level shape: 0 lines.
Ops/logs-memexcloud-20260924-verify-5388-stall, |~ "(?i)Thread Pool execution stalled|Grain placement operation timed out": cut at 200. The landed sample holds dozens of sub-10 s stalls on ci.9262 and ci.9263 pods, for example:
- 8.64 s on 6f497dc9b-r4nkg
- 6.51 s on 6f497dc9b-j9vq8
- 5.00 s on 6f497dc9b-hd5jg
- 3.0–3.7 s repeatedly on 7d7f4d84f9-mvzb5
- 1.39 s on 76cf776847-9p7z4 (19:02Z)

Reading. The fingerprint as filed (≥10 s, Error) has not recurred in 15 h. That is not a clean reading, though. The same gauge fires at 1–9 s, many times, on the fixed image, and the separate placement query counts 177 Grain placement operation timed out lines in the same window (#5037). This ticket says the gauge is a symptom, and the symptom has only dropped below the Error threshold. Not closing on a threshold.

Not established. The cause of the residual stalls: GC (#4824 / #5555) or CFS throttling. Nor whether ci.9287 (core bf5ac85526, which carries #5635's removal of the same-silo deep copy) changes the rate. ci.9287 has been handed to the control lane but has not rolled.

Stall readings vs GC on memex-cloud, and two blockers removed in #5678

I ran Logs InstanceActions on memex.meshweaver.cloud (Ops/logs-memexcloud-20260925-5388-*). Each stall writes 2 lines: the stall line and Self-monitoring … degraded.

1. Rate per image

window	images (core)	stall readings	longest
09-23 14:38–17:49Z (3.2 h)	ci.9258–9262 (`9a61fed964`)	~500. The 48 h read (`…-stalls-48h`) was CUT at 1000 lines inside this window. I parsed 204 of them; 21 were ≥ 5 s	10.16 s
09-24 06:58–08:13Z (a roll)	ci.9263 survivors (`76cf776847`) and the next ReplicaSets	26 (`…-stalls-from-0645`)	3.77 s
09-24 08:41Z – 09-25 04:55Z (20.2 h)	ci.9291 (`140c5fb7ae`, 4/4 replicas per `Ops/Status/memex-cloud` v3788)	4 (`…-stalls-all`, not cut)	1.48 s

The ">200 per ~15 h of 1–9 s" family belongs to the 9a61fed964 images. The current image logs about one sub-1.5 s reading every 5 h.

2. GC or blocking on the current image

The [LIVENESS] heartbeat runs on a dedicated thread, so a GC or a machine pause shows up in it as OVERRAN.

…-overran (last 20 h): 0 OVERRAN ticks on any pod.
Positive control: the same query over 48 h (…-overran-48h) finds 21.

For the three stalls on 589ff8f895-ghgjt (…-liveness-ghgjt-windows), GC pause measured in the 10 s tick that holds each stall:

stall	tick	gcPause in tick	gen2	poolThreads / pending
13:23:18Z, 1.05 s	1680	+1.25 s	+1	16 / 0
14:16:05Z, 1.40 s	1996	+0.82 s (+0.48 s in 1995)	+0 (+1)	10 / 1
19:29:49Z, 1.42 s	3876	+0.03 s	+0	10 / 0

2 of the 3 are GC pauses. The heap behaviour behind them is MessageService DrainLoop throws OutOfMemoryException during hub delivery (recurrence of #4824) #5555's subject.
1 is genuine pool starvation at a quiet moment. The heartbeat was on time (gap 10.00 s), so it was not a machine pause either. Nothing I can read names what held the workers.

3. Static sweep: what can block pool threads

Core and Plugins (origin/main) contain no .Result, .Wait(), .GetAwaiter().GetResult() or Thread.Sleep on a hot path. The existing ratchets hold src/ at zero, apart from the tool-only allow entries. What the sweep did find is structural.

Every IIoPool's InvokeBlocking leaves ran on borrowed ThreadPool workers.
- The caps are FileSystem 256, Blob 128, Http 16, and the core count for Compile and the default pool. The pool minimum is 6.
- A burst of module-volume walks, bundle reads, git waits or prebuilt reads could therefore hold every worker.
- Only CompileCpu had threads of its own.
- The guards could not see this: the call site was correct, and the pool did the borrowing.
A kernel code cell's Roslyn compile ran on a pool worker. It ran as one async Compile-pool leaf with ConcurrentBuild on, which is the fix(compiler): keep NodeType compiles off the silo's shared ThreadPool #5327 shape.

#5678 fixes both.

Every pool's blocking leaves now run on threads the pool starts itself: mw-io-lane, with the same cap and the same queue.
ScriptSession.Compile is now a CPU-lane leaf with ConcurrentBuild off.
Tests with negative controls: BlockingLeavesStayOffTheThreadPoolTest, KernelCellCompileStaysOffTheThreadPoolTest.
Write-up: Doc/Architecture/BlockingLeavesOffTheThreadPool.

Candidates in Plugins, not fixed here:

MeshWeaver.Blazor/Pages/ContentPage.razor does a synchronous CopyTo/ReadToEnd of a content stream during render.
MeshWeaver.Azure.Blob/BlobNuGetPackageCache runs ZipArchive over a blob stream inside an async Run lane.
MeshWeaver.Import/ImportBuilder parses Excel synchronously inside an async Run lane.

All three are user- or Azure-triggered. None of them was observed in these windows.

Not established

What held the pool at 19:29:49Z on ghgjt.
Which change between 9a61fed964 and 140c5fb7ae produced the drop. Candidates: fix(mesh): a subscriber's own release is not announced back; a same-silo grain call shares the immutable delivery envelope #5635 (no same-silo JSON deep copy), fix(compile): stop dual-writing the compile-state satellite nobody reads (#5389) #5571, fix(query): a live query re-reads once per burst of changes, not once per change #5615 and Plugins#2329. The windows also differ in load: 09-23 was a day of rolls and heap steps.

What closes this

After a roll carrying #5678: the same stall query per pod over ≥ 24 h, with each reading checked against its tick's gcPause. Any remaining stall with gcPause ≪ stall is starvation that still needs a name. A GC-paired stall belongs with #5555.

_{Claude Opus 5.5. The only production writes were Logs InstanceActions under Ops/logs-memexcloud-20260925-5388-* on memex-cloud.}

review: the cell's execute leaf returns the task directly; the IoPool…

#5678 merged to main (96acfe24a6, 2026-09-25 07:05Z). It is not live until a roll carries it. The verification after that roll is in the comment above. This issue stays open until that reading is taken.

Deploy status, 2026-09-26 ~06:10Z:

fix(iopool,kernel): blocking leaves and a cell's compile run off the ThreadPool (Refs #5388) #5678 (96acfe24a6) is not live on memex. memex runs 4028fd6468 (ci.9332), and that image does not contain it.
memex-cloud is mid-roll onto ci.9349 (7bc7ab26f0), which does contain it. Three generations were running and converged: false at 06:01Z.

The verification defined in the 09-25 comment applies once that roll converges.

Separate finding, so it is not read as this issue. Since ~09-25 18:00Z, memex pods have been dying abruptly (tracked on #5704). There are SIGABRT/SIGSEGV dumps, and there are also deaths with no log line at ~3.03 h of uptime. Their last [LIVENESS] ticks are on time (gap=10.00s) with poolPending=0, so those deaths are not ThreadPool stalls.

Still occurring: 1 more since the last update (7 total).

Last seen: 2026-10-05 07:43:40Z
Pods: memex-portal-deployment-857546649-2c25b, memex-portal-deployment-6bf5d7dcc8-jl5k8, memex-portal-deployment-79889596-47trv, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-6d8c9665cc-jltnc

Recent log lines

2026-09-23 13:13:23Z memex-portal-deployment-79889596-47trv fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
      .NET Thread Pool execution stalled for 11.3939403s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.
2026-09-23 14:51:41Z memex-portal-deployment-777677694-p6hbn fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
      .NET Thread Pool execution stalled for 10.1619648s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.
2026-10-05 07:43:40Z memex-portal-deployment-6d8c9665cc-jltnc fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
      .NET Thread Pool execution stalled for 17.1919705s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.

_{Re-addressed by the current identity function: this incident inherited 3a7d53ee7da8a1b4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Still occurring: 1 more since the last update (8 total).

Last seen: 2026-10-05 16:19:25Z
Pods: memex-portal-deployment-857546649-2c25b, memex-portal-deployment-6bf5d7dcc8-jl5k8, memex-portal-deployment-79889596-47trv, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-6d8c9665cc-jltnc, memex-portal-deployment-6cfcf78898-gkkdv

Recent log lines

2026-09-23 14:51:41Z memex-portal-deployment-777677694-p6hbn fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
      .NET Thread Pool execution stalled for 10.1619648s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.
2026-10-05 07:43:40Z memex-portal-deployment-6d8c9665cc-jltnc fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
      .NET Thread Pool execution stalled for 17.1919705s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.
2026-10-05 16:19:25Z memex-portal-deployment-6cfcf78898-gkkdv fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
      .NET Thread Pool execution stalled for 21.4995069s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.

_{Re-addressed by the current identity function: this incident inherited 3a7d53ee7da8a1b4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Heartbeat reading for each named occurrence: 09-22 18:47 2c25b, 09-23 12:30 jl5k8, 13:13 47trv, 14:51 p6hbn, and 10-05 07:43 jltnc / 16:19 gkkdv. All six are on GC-thrash pods: heap 10.2–11.6 GiB, GC 86–97 % of wall time, except 47trv, where the stall tick held a single 11.7 s GC. The Thread Pool execution stalled signal here is the GC, not a scheduler defect. This closes with the retention fix (#6261 + Plugins#3109, not yet rolled). I did not read the comment-only timestamps (09-19 06:29, 09-23 10:02, 09-24 06:51, 09-25 07:05), because their pods are not named. Table: #5315 (comment)

Post-roll reading, 2026-10-10 ~05:20Z. All three portals serve core 479c9b72d5 (/api/version), which carries #6261 (merged 2026-10-07 17:06Z). Plugins#3109 merged 2026-10-07 17:57Z, before any image in the windows below was built.

Governed Logs reads on the control instance (Ops/Actions/logs-*-20261010-*):

	memex-cloud (10-09 17:15Z → 10-10 05:15Z)	memex (10-09 05:15Z → 10-10 05:15Z)
Pod boots in window (`warm-up complete`)	34, across 11 ReplicaSets	29, across 10 ReplicaSets
`Thread Pool execution stalled`	13 lines, longest 2.44 s (11 of 13 within 35 min of their pod's boot)	5 lines, longest 1.93 s
`Thread Pool execution stalled for NN` (≥ 10 s)	0	0
`DISPOSAL DEADLOCK DETECTED` (`MessageHub[7316]`/`[7317]`)	0	0
any `MessageHub[73xx]` or `[DISPOSE-*`	0	0
`[LIVENESS]` tick with heap ≥ 6 GiB on the CURRENT ReplicaSet	0 (`d9cd97`, since 22:34Z; its `[HEAPSTEP]` lines peak at 5.06 GiB)	0 (`477bf6`, since ~23:00Z; `[HEAPSTEP]` peak 3.83 GiB)

For comparison, the stall band this issue tracked ran at heap 10.2–11.6 GiB with GC at 86–98 % of wall time. The heaviest tick on a #6261 image in either window is memex cfb58787b-b6klj on 10-09 14:35Z: heap 8.09 GiB, but gcPause +0.19 s per 10 s tick (~2 %) and poolPending=0. The heap is high there, but the GC is not thrashing.

Positive controls. (1) The pool-stall instrument fires in the window: 18 sub-3 s lines, so a ≥ 10 s stall would have printed. (2) [LIVENESS]/[HEAPSTEP] are ingested from the current pods. (3) The same heap regex returns 255 ≥ 6 GiB ticks from the pre-#6261 6548d4 pods (booted ~10-07 13:15Z, before the merge) up to 10-09 17:58Z. That control separates the images.

Not established. Loki keeps under ~4.5 days. A probe for any line from the 10-05 pod 6cfcf78898-kss8q returned 0, so the 10-05 episode itself cannot be re-read as an in-window control for the DISPOSAL DEADLOCK regex. That regex is the literal text at MessageHub.cs:3084/3116/3145. The windows are 12 h and 24 h of ordinary traffic plus frequent rolls. They are not a reproduction of the 10-05 surge.

Closing. The stated closing condition was "closes with the retention fix (#6261 + Plugins#3109)". The ~25 s and 10–17 s LocalSiloHealthMonitor stalls are gone from both portals. The only stalls left are 1–2.4 s, mostly in a pod's first half hour, which is warm-up compilation, not GC thrash. Reopen if a stall of 5 s or more recurs on a pod whose [LIVENESS] line shows heap under 6 GiB.

_{sev:M sweep, Claude Opus 5.5 (agent session).}

Navigation Menu

Repository navigation

Recurrence: Orleans silo thread-pool stall (~25s) in memex-portal — continued from #4886, risk of missed heartbeats and cluster eviction #5388

Evidence

Triage 2026-09-23: same site and discriminator as #4886. Its one occurrence predates the fix candidate, so only the post-roll read remains

Canonical for the `LocalSiloHealthMonitor` stall site — #4886 folded here

Post-roll reading 2026-09-24 ~06:51Z (memex-cloud): the Error-level stall is quiet, the starvation it gauges is not. Stays open

Stall readings vs GC on memex-cloud, and two blockers removed in #5678

1. Rate per image

2. GC or blocking on the current image

3. Static sweep: what can block pool threads

Not established

What closes this

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Recurrence: Orleans silo thread-pool stall (~25s) in memex-portal — continued from #4886, risk of missed heartbeats and cluster eviction #5388

Description

Evidence

Activity

rbuergi commented on Sep 23, 2026

Triage 2026-09-23: same site and discriminator as #4886. Its one occurrence predates the fix candidate, so only the post-roll read remains

systemorph-com commented on Sep 23, 2026

systemorph-com commented on Sep 23, 2026

rbuergi commented on Sep 23, 2026

Canonical for the LocalSiloHealthMonitor stall site — #4886 folded here

rbuergi commented on Sep 24, 2026

Post-roll reading 2026-09-24 ~06:51Z (memex-cloud): the Error-level stall is quiet, the starvation it gauges is not. Stays open

rbuergi commented on Sep 25, 2026

Stall readings vs GC on memex-cloud, and two blockers removed in #5678

1. Rate per image

2. GC or blocking on the current image

3. Static sweep: what can block pool threads

Not established

What closes this

rbuergi commented on Sep 25, 2026

rbuergi commented on Sep 26, 2026

systemorph-com commented on Oct 5, 2026

systemorph-com commented on Oct 5, 2026

rbuergi commented on Oct 8, 2026

rbuergi commented on Oct 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Canonical for the `LocalSiloHealthMonitor` stall site — #4886 folded here