Repository navigation
Orleans silo thread-pool stalls (12–18s) across all memex-portal pods risk missed heartbeats and cluster eviction #4886
Description
What is failing
The Orleans silo in the memex-portal deployment is experiencing full .NET Thread Pool execution stalls of 12–18 seconds, detected by Orleans.Runtime.MembershipService.LocalSiloHealthMonitor. During such a stall the silo cannot run any tasks — not even timer callbacks — which puts it at risk of missing membership heartbeats and being marked dead/evicted by the cluster.
Probable cause
Cause unclear from the log evidence alone — the monitor reports the symptom, not the trigger. Plausible candidates, roughly in order of likelihood:
- Very long GC pauses (e.g. a full blocking Gen2/LOH collection triggered by a large object allocation burst or high memory pressure).
- Thread pool starvation — more concurrent blocking work items than available threads, so the pool ramps up slowly (the 12–18s duration is consistent with a slow ramp on a small container CPU limit).
- CPU throttling / CPU-quota cgroup exhaustion on the pod, freezing all runtime execution.
No exception or application stack was captured, so a first step should be correlating the three stall timestamps with GC stats, pod CPU-throttling metrics (container_cpu_cfs_throttled_periods_total), and memory pressure.
Impact
Low frequency so far — 3 occurrences over 4 days (first 2026-09-15T23:45Z, last 2026-09-19T08:57Z), each on a different pod of the memex-portal deployment. That it hits every pod at least once suggests it is systemic to the workload/deployment rather than one unhealthy instance. No outage has been observed, but a stall of this length in a silo hosting grains can cause transient request timeouts and, if it lengthens past the membership-death-refresh window, cluster eviction and grain migration churn. Worth investigating before it becomes the failure mode of a real incident.
Where to look
- Runtime/membership diagnostics:
Orleans.Runtime.MembershipService.LocalSiloHealthMonitor(Orleans framework silo health watchdog) — the stall detection and logging site. - Deployment config: CPU/memory limits of the
memex-portal-deploymentpods — check CPU-limit throttling at the three stall timestamps. - App-side heavy allocations that could trigger Gen2/LOH collections during the stall windows.
Evidence
| Fingerprint | 1dbb05f503932ad3 |
| Category | Orleans.Runtime.MembershipService.LocalSiloHealthMonitor |
| Severity | Error |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-58dbb6d98d-8msww, memex-portal-deployment-7fd9668b69-sjm5v, memex-portal-deployment-69956b6dbc-gj8m2 |
| Occurrences | 3 |
| First seen | 2026-09-15 23:45:48Z |
| Last seen | 2026-09-19 08:57:04Z |
Recent log lines
2026-09-15 23:45:48Z memex-portal-deployment-7fd9668b69-sjm5v fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
.NET Thread Pool execution stalled for 14.3444308s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.
2026-09-19 06:29:21Z memex-portal-deployment-58dbb6d98d-8msww fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
.NET Thread Pool execution stalled for 12.6214793s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.
2026-09-19 08:57:04Z memex-portal-deployment-69956b6dbc-gj8m2 fail: Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]
.NET Thread Pool execution stalled for 18.3500617s. This can indicate .NET Thread Pool starvation, very long .NET GC pauses, or other runtime or machine pauses.
Re-addressed by the current identity function: this incident inherited 3a7d53ee7da8a1b4 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/1dbb05f503932ad3. Recurrences are folded into this issue rather than opening new ones.
Activity
systemorph-com commentedon Sep 19, 2026
Still occurring: 1 more since the last update (4 total).
memex-portal-deployment-58dbb6d98d-8msww,memex-portal-deployment-7fd9668b69-sjm5v,memex-portal-deployment-69956b6dbc-gj8m2,memex-portal-deployment-69956b6dbc-gx6z6Recent log lines
Re-addressed by the current identity function: this incident inherited
3a7d53ee7da8a1b4(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 20, 2026
Still occurring: 1 more since the last update (5 total).
memex-portal-deployment-58dbb6d98d-8msww,memex-portal-deployment-7fd9668b69-sjm5v,memex-portal-deployment-69956b6dbc-gj8m2,memex-portal-deployment-69956b6dbc-gx6z6,memex-portal-deployment-69956b6dbc-ndnxtRecent log lines
Re-addressed by the current identity function: this incident inherited
3a7d53ee7da8a1b4(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 21, 2026
One of the three occurrences is now attributed: it is the same event as #4824
Grouped with the
RoutingGrainsaturation cluster — the full grouping and its evidence are in#5018 (comment). This ticket is kept
as its own defect: it is the independent measurement of the quantity the routing back-pressure
line only ever infers, and the routing tickets are right to point here rather than absorb it.
Occurrence 2 is a full blocking GC, not ThreadPool starvation
.NET Thread Pool execution stalled for 12.6214793s— podmemex-portal-deployment-58dbb6d98d-8msww, 2026-09-19 06:29:21ZOutOfMemoryExceptioninObjectPolymorphicConverter.Writeat 06:28:25Z and 06:28:52Z, then inIoPool.SubscribeThroughPoolCoreat 06:30:09Z and 06:31:04ZThe stall sits inside that heap-exhaustion window, on the same pod. A 12.6 s whole-runtime
stall in the middle of an OOM burst is the first candidate this ticket itself lists — "very long
GC pauses (e.g. a full blocking Gen2/LOH collection triggered by a large object allocation burst or
high memory pressure)". For this occurrence that is now evidenced rather than merely ranked.
It also answers this ticket's own next step ("correlate the three stall timestamps with GC stats,
pod CPU-throttling metrics and memory pressure") for one of the three, out of evidence already in
the tracker and with no pod access needed: the memory-pressure correlate is #4824.
The other two stay unattributed, and this ticket stays open on them
7fd9668b69-sjm5v, 14.3 s69956b6dbc-gj8m2, 18.4 sNeither has an OOM correlate anywhere in the tracker, so CPU-quota throttling and genuine thread-pool
starvation both remain open for them.
What this ticket is NOT
🚨 It is not downstream of "routing slots that never release" — the arrow points the other way.
Every multi-sample reading at the routing back-pressure log site shows episode stamps rising on the
same activation, which by that site's own rule means the episode drained; slots are released. The
routing line has the direction right: "a CPU-starved silo raises this with nothing stuck." So a
stall measured here is upstream of the routing saturation reports on these pods, not caused by
them.
🚨 #5037 is not a duplicate of this ticket and should not be closed as one. Different log site,
different namespace (
memexvsmemex-cloud), different days — and there is no stall measuredhere inside #5037's window (the latest occurrence here is 2026-09-19 08:57Z; #5037's burst is
2026-09-20 22:25:14Z). Same family — Orleans runtime work that cannot complete on a stalled pool —
but not the same event, so folding them would assert a correlation nothing measured.
fix(routing): the saturation report prints the gauge that can see the…
rbuergi commentedon Sep 22, 2026
Grouped with #2480 / #2833 / #4847 / #4888 — the full grouping is on #2480
Summary of where this one sits, and why nothing here was fixable from the code.
This is a symptom GAUGE, not a trigger.
LocalSiloHealthMonitorreports that the pool stalled;it names no cause, and a stall gauge rises on any resource fault — so reading it as the cause of
whatever fired next would invert the arrow, which is a mistake this estate has already paid for.
What the co-occurrence evidence actually is, with its denominator. Exactly one pod carries both
this and #2480:
memex-portal-deployment-69956b6dbc-gj8m2, an 18.35 s stall at 2026-09-1908:57:04Z followed by an
IoPoolSiloTeardown30 s join expiry at 08:59:41Z. The other four #2480samples have no stall reading on their pod and two of this issue's three stalls have no expiry on
theirs — 1 of 5 and 1 of 3. Correlation present, nowhere near universal, direction unestablished.
Why it matters anyway: a 12–18 s stall inside a 30 s join is over half the budget, so while it is
happening any teardown budget can expire with no leaf "ignoring" anything — which would make #2480's
own log sentence wrong for that occurrence. The discriminator is on #2480.
The readings that would identify the trigger, none of which I took:
container_cpu_cfs_throttled_periods_totalon those pods in those windows (candidate 3);ThreadPool.ThreadCount/PendingWorkItemCounton the pod (candidate 2, and the same readingHosted sync hub's disposal drain starves on an exhausted thread pool (M1-by-pool-starvation, post-#3593) #4847 asks for).
I am not guessing which of the three it was; the three have different owners and different fixes.
One thing that HAS moved and is worth knowing before the next reading: #5182 (merged
2026-09-22, not yet rolled) bounds
MessageService.DrainLoop's trampoline at 64 synchronous turns.That trampoline — a hub drained until empty owning a pool worker for an unbounded run — is #4847's
named candidate for what saturates the pool, and therefore a candidate contributor here. If these
stalls recur on an image carrying
358ec858aa, that candidate is excluded rather than suspected.rbuergi commentedon Sep 23, 2026
Duplicate of #5388 — same log site and quantity:
Orleans.Runtime.MembershipService.LocalSiloHealthMonitor[86528487]".NET Thread Pool execution stalled for …", namespace memex-cloud. #5388 is kept as the canonical because it is the issue the live incident node (Admin/_LogIncident/eda900bec4354ee0, which re-addressed this ticket's1dbb05f503932ad3) now folds recurrences onto — closing it instead would be reopened by the recurrence bot. This ticket's analysis (occurrence 2 = a full blocking GC inside #4824's OOM window; symptom gauge, not a trigger; grouping on #2480) is linked from #5388.