Repository navigation
Healthy Orleans silo falsely voted dead by peers and terminated itself during cluster roll-churn #6395
Description
What is failing
A live, actively heartbeating Orleans silo in the memex portal cluster was marked Dead in the membership table by two peer silos and honored the verdict by terminating itself (MembershipTableManager.KillMyselfLocally → FatalErrorHandler). The silo's last IAmAlive update postdates all three suspicion votes, so it was alive and writing to the membership table at the moment it was voted dead — a false-positive death declaration, not a genuine crash or rolling-shutdown race (the pod was not terminating; the silo had started minutes earlier).
Probable cause — medium confidence
The suspicion protocol (default: 2 distinct silo votes) fired because the target silo failed to answer direct probes from both peers within the freshness window, even though its table-written heartbeats continued. That asymmetry typically points to transient unreachability of the pod — CPU/threadpool starvation, a long GC pause, or a node-level network hiccup — rather than a dead process. The wider cluster was under heavy roll-churn at the same moment (see the SiloUnavailableException roll-churn bursts ending seconds before this event), so deployment churn / load is the likely trigger. The precise reason the probes were missed is not determinable from these logs — it needs pod resource metrics for the implicated pods around the event.
Impact
Single occurrence on a single pod at the time of this incident. One healthy portal pod killed itself; in-flight grain calls to it failed transiently (the companion SiloUnavailableException burst at the same moment), and the pod was restarted by Kubernetes. User impact was transient, but a membership protocol that kills healthy silos under load amplifies instability during deployments.
Where to look
Orleans.Runtime.MembershipService.MembershipTableManager—CheckIfLocalSiloIsDead/KillMyselfLocally(/_/src/Orleans.Runtime/MembershipService/MembershipTableManager.cs), and the peer-side suspicion path (ClusterHealthMonitor/ vote accumulation) on the suspecting silos.- Pod resource and GC logs for the terminated pod and its two suspecting peers around the event window.
- Related incidents: the
FatalErrorHandlercompanion atAdmin/_LogIncident/bfcac5a35d2571b4(same event, carries the full stack), and the ongoing roll-churn family e.g.Admin/_LogIncident/roll-churn-1e98b289db877f91— this false-positive kill may be another symptom of that churn rather than an independent defect.
If investigation shows this only occurs during roll-churn windows, it may be worth tuning ClusterMembershipOptions (probe timeouts / vote thresholds) for this deployment instead of chasing a code fix.
Evidence
| Fingerprint | 89dfb3b5f652b720 |
| Category | Orleans.Runtime.MembershipService.MembershipTableManager |
| Severity | Error |
| Namespace | memex |
| Pods | memex-portal-deployment-5f6b9fc9b-hrdhq |
| Occurrences | 1 |
| First seen | 2026-10-09 20:44:12Z |
| Last seen | 2026-10-09 20:44:12Z |
| Routing | not determined — no configured route matches the category Orleans.Runtime.MembershipService.MembershipTableManager. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-10-09 20:44:12Z memex-portal-deployment-5f6b9fc9b-hrdhq fail: Orleans.Runtime.MembershipService.MembershipTableManager[100627]
I have been told I am dead, so this silo will stop! Reason: I should be Dead according to the membership table (in Refresh). Local entry: [SiloAddress=S10.244.17.110:11111:150582958 SiloName=Silo_9dc7e Status=Dead HostName=memex-portal-deployment-5f6b9fc9b-hrdhq ProxyPort=30000 RoleName= UpdateZone=0 FaultZone=0 StartTime=2026-10-09 20:35:58.118 GMT IAmAliveTime=2026-10-09 20:44:06.335 GMT Suspecters=[S10.244.3.225:11111:150580999, S10.244.2.180:11111:150580798, S10.244.3.225:11111:150580999] SuspectTimes=[2026-10-09 20:43:48.767 GMT, 2026-10-09 20:43:48.767 GMT, 2026-10-09 20:43:48.767 GMT]].
Opened automatically from Admin/_LogIncident/89dfb3b5f652b720. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 3ce5feb1528fd58f: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
Widen Orleans membership probe tolerance for MeshWeaver silos (Refs #…
Merge pull request #6428 from Systemorph/bugfix/systemorph-meshweaver…
rbuergi commentedon Oct 10, 2026
Why the probes went unanswered: the silo never worked, and the Dead verdict was correct. No thread-pool stall.
I read the window with governed
Logsactions on the control instance, all read-only. Each one names its LogQL, its window and its line count on the node, underOps/Actions/w8d-6395-*:w8d-6395-health-hrdhqw8d-6395-killed-joinw8d-6395-timeouts-by-podResponse did not arrive on timew8d-6395-monitor-ptszc845898d964-ptszc)w8d-6395-probes-narrowWhat the lines establish
The silo failed from 33 s after it started, not at 20:43. It started at 20:35:58. Its first unanswered request was logged at 20:36:31.329:
Request [S10.244.17.110 … sys.client/hosted-10.244.17.110]->[S10.244.17.110 … messagehub/Agent/_Activity/import-manifest] … DeliverMessage … About to break its promise. That is a call from the silo's own hosted client to a grain on the same silo, sent around 20:36:01. From then until the kill it got no answer to anything. That covers the static-repo import activations,GetMeshNode(10 s budget),CreateNode(LifecycleVerdictStalledException) and thesys.svc.cancelercalls to10.244.3.24.Only this pod. Across the whole namespace, 20:35:30–20:38:30, 71 of 71
Response did not arrive on timelines come from5f6b9fc9b-hrdhq. No other replica, old or new generation, logged one. The rest of the cluster was healthy while this pod was not.Its thread pool was not stalled when the probes failed. Orleans' own
LocalSiloHealthMonitoron the pod reported a non-zero score exactly three times in the window:.NET Thread Pool execution stalled for 1.276s;… stalled for 1.098s;This silo is not active (Status: Dead), which comes after the verdict.Between 20:38:46 and the kill the monitor reported nothing, which means a zero score. A zero score covers both its thread-pool check and its "received no probe request" check. The pod logged no GC, starvation or long-turn line in the window.
The monitors' side was plain silence.
ptszcloggedRequested probe timeout 00:00:07.55 exceeded→TimeoutException: The probe exceeded its timeouton each direct probe. Indirect probes via10.244.3.225failed the same way (Indirect probe request #37 … failed after 00:00:07.53 with a direct probe response time of 00:00:07.53). The probes and the votes logged no exception, connection refusal or reset. Messages were not refused. Their answers never came back.Reading
This is not "a healthy silo killed under load". From its first minute the silo could not complete a single request, including calls to grains it hosted itself. Those calls need its grain-directory registration on other silos. Meanwhile its local health checks were green, and it was receiving probe requests. That fits a broken outbound/response path from this one pod: requests reach it, nothing it sends back arrives. It does not fit a blocking turn or a starved scheduler, because the thread-pool monitor rules those out for 20:38:46 → 20:44. Membership evicting it after about 8 minutes of total dysfunction was the right outcome.
What I did NOT establish
w8d-6395-killed-connmatched 785 lines, and of the 43 whose text names a connection none isConnectionFailedExceptionor a reset. They are the node-kernel and python sidecars' localhost connect failures, which have their own story. I took no node or network reading. The pod ran onaks-silos-10171001-vmss000028. That node also hosted65f6466887-dfvm7on 2026-10-07, whose lines in an unrelated query showConnection reset by peer. That is one coincidence, not evidence about the node.5f6b9fc9bstarted cleanly. Point 2 says they logged no timeouts in the first three minutes. I did not read them past that.On #6428 (already closed): keep it closed
It widened
ProbeTimeoutto 10 s on the premise that "the repository configured noClusterMembershipOptions". That premise does not hold for the fleet:Memex.Portal.Distributed(MeshWeaver.PluginsProgram.cs) already setsProbeTimeout = 15 s,NumMissedProbesLimit = 5and indirect probes (see #5300's triage). The measured probe budget above, 7.5 s per probe, is that configuration. More importantly, the silo here did not need more tolerance. It answered nothing for 8 minutes, so a wider window would only have kept a dead member in the table longer. Every caller routed to it would have waited out 30 s timeouts, which is the cluster-wide cost #5300 and #2906 describe.No code change from this read: the root cause is outside the process, as far as the logs reach. If it recurs, take a
Sampleof the new pod during its first minute, plus its node, while it is still alive.w8d, Claude Opus 5.5 (agent session). Read-only.
rbuergi commentedon Oct 10, 2026
Revision of the reading above. Detail on #6432: #6432 (comment)
The 2026-10-10 recurrences (5j9gx, gvt4n) and a second look at hrdhq point inside the process, not at the pod's outbound network path:
messagehub/Agent/_Activity/import-manifestplaced on the same silo. That is 85 of 85 such lines on memex 20:20–20:45 (all hrdhq) and 52 of 52 on memex-cloud 08:05–08:56 (all 5j9gx and gvt4n). The healthy new pods in the same rolls created their import manifest in about 3 s.poolThreadsclimbs from 20 to 55 over 16 min while it completes about 500 work items per 10 s. A healthy pod in the same roll runs on 16 threads at about 47 000.SiloConnection.HandlePingMessage). Everything else goes throughMessageCenter.ReceiveMessageand grain activation on that loop. A silo whose activation or dispatch path blocks therefore keeps passing probes for minutes while answering nothing else. That explains the 8 to 20 minutes to eviction without any connection error.The Dead verdict was still correct, and the conclusion against #6428-style tolerance stands. The blocking call itself still needs the FailFast heap dumps (5j9gx 08:30:03Z, gvt4n 08:54:10Z on memex-cloud), which need cluster access.
w11a, Claude Opus 5.5 (agent session). Read-only.
docs(silo-eviction): one ping mechanism, the options that were deploy…
docs(silo-eviction): a self-kill may leave no dump; IAmAlive is not e…
rbuergi commentedon Oct 11, 2026
Closing as a duplicate of #6432: same defect, confirmed from the evidence on both issues.
5f6b9fc9b-hrdhq, memex, 2026-10-09) carries the same boot fingerprint as the three memex-cloud evictions tracked on Silo 5j9gx marked Dead by peers 26s after its last heartbeat and terminated itself — recurrence of the false-positive membership kill (#3053) #6432: 85 of 85did not arrive on time … _Activity/import-lines in its roll window are its own, starting 33 s after the silo started (comments of 2026-10-10 above).Ops/Actions/analyzedump-memex-cloud-20261011-ltlwc-2): the root container's single-instance lock is held by one thread for the life of the process, with 21 threads waiting on it, among them Orleans' own construction of the codec forIndirectProbeResponse. A silo in that state cannot answer a probe. The reading is on Silo 5j9gx marked Dead by peers 26s after its last heartbeat and terminated itself — recurrence of the false-positive membership kill (#3053) #6432: Silo 5j9gx marked Dead by peers 26s after its last heartbeat and terminated itself — recurrence of the false-positive membership kill (#3053) #6432 (comment)So neither "roll churn" nor probe tolerance is the subject here, and the verdict on this silo was correct. What is still open (which call holds that lock) is one question with one next reading, and it is tracked on #6432 so that it is not answered twice.
w21, Claude Opus 5.5 (agent session).