Repository navigation
Orleans grain directory serves stale silo address after restart, briefly rejecting message-hub deliveries #5336
Description
What is failing
The Orleans client on the portal pod (memex-portal-deployment-5c444645f8-8pdzb, memex-cloud) failed to deliver IMessageHubGrain.DeliverMessage requests to message-hub grain targets (e.g. messagehub/AgenticBusiness, messagehub/Collaboration/_GitSync, messagehub/Essentials/_Activity/import-manifest). Message placement failed because the client could not open a connection to the destination silo S10.244.9.90:11111:149090968; the messages were rejected with OrleansMessageRejectionException wrapping ConnectionFailedException.
Probable cause
Stale silo address in the grain directory / membership view (medium confidence). The destination silo's generation in the failing address (149090968) is older than the portal client's own generation (149091044) — the client was trying to reach a superseded silo activation whose address was still returned by LocalGrainDirectory.LookupAsync (stack runs through DhtGrainLocator.Lookup → PlacementService.PlacementWorker). All 28 failures landed within an ~8.6 s window and then stopped, consistent with directory convergence catching up after a silo restart rather than a persistent network partition. Not certain: if silo 10.244.9.90 was genuinely down at that moment, this is the transient expected during a rolling restart — check whether that silo restarted around 2026-09-22 14:41 UTC.
Impact
Minimal: 28 rejected messages over ~8.6 s on a single pod, no recurrence. Message delivery retries after the connect backoff (sub-second), so the user-visible effect is likely nil.
Where to look
MeshWeaver.Connection.Orleans— theIMessageHubGraindelivery path and how the portal client resolves message-hub grain placements.- Orleans runtime internals in the stack:
Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync,Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync,Orleans.Runtime.Placement.PlacementService. - Check memex-cloud silo
10.244.9.90restart/deployment events around 2026-09-22T14:41Z to confirm or refute the stale-directory hypothesis.
Evidence
| Fingerprint | 445e1378ed560f40 |
| Category | Orleans.Messaging |
| Severity | Error |
| Exception | Orleans.Runtime.OrleansMessageRejectionException |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-5c444645f8-8pdzb |
| Occurrences | 28 |
| First seen | 2026-09-22 14:41:30Z |
| Last seen | 2026-09-22 14:41:38Z |
| Routing | not determined — no configured route matches the category Orleans.Messaging. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-22 14:41:30Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/AgenticBusiness] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B484574
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.9.90:11111:149090968, will retry after 977.9175ms
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 129
at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 247
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 902
at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 540…[truncated]
2026-09-22 14:41:30Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/Collaboration/_GitSync] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B484573
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.9.90:11111:149090968, will retry after 846.6106ms
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 129
at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 247
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 902
at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:l…[truncated]
2026-09-22 14:41:30Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/Essentials/_Activity/import-manifest] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B484576
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.9.90:11111:149090968, will retry after 814.2222ms
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 129
at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 247
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 902
at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/Placeme…[truncated]
2026-09-22 14:41:38Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/AppleIntelligence] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B485B35
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.9.90:11111:149090968, will retry after 652.1773ms
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 129
at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 247
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 902
at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 5…[truncated]
2026-09-22 14:41:38Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/Feedback/Feedback] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B485B0F
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.9.90:11111:149090968, will retry after 640.5595ms
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 129
at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 247
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 902
at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 5…[truncated]
2026-09-22 14:41:38Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/Hosting/RepoHealth/_Activity/compile-state] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B485BF8
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.9.90:11111:149090968, will retry after 636.1279ms
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 129
at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 247
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 902
at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/P…[truncated]
2026-09-22 14:41:38Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/Hosting/InstanceRequest] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B485BF7
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.9.90:11111:149090968, will retry after 178.318ms
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 129
at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 247
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 902
at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:l…[truncated]
2026-09-22 14:41:38Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/AgenticPrimer/DiceGame] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B485AB8
Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.9.90:11111:149090968, will retry after 163.101ms
at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 129
at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 247
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 902
at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 35
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:li…[truncated]
Re-addressed by the current identity function: this incident inherited f367c5512327cc57 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/445e1378ed560f40. Recurrences are folded into this issue rather than opening new ones.
Activity
systemorph-com commentedon Sep 22, 2026
Still occurring: 17 more since the last update (45 total).
memex-portal-deployment-5c444645f8-8pdzbRecent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 23 more since the last update (51 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22dRecent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 22, 2026
Still occurring: 128 more since the last update (179 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22dRecent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 23, 2026
Triage 2026-09-23: canonical ticket for the 2026-09-22 14:41Z silo departure on memex-cloud. Six siblings folded in
Seven issues come from one event: silo
S10.244.9.90:11111:149090968left. Its generation dates its start to 14:09:28Z. The survivors were5c444645f8-8pdzb(started 14:10:44Z) and89868db67-xg22d. The second is a new ReplicaSet, started 14:40:46Z, so a roll had begun. Every sample falls between 14:41:30 and 14:42:06Z on those two pods, and none has a MeshWeaver frame.Orleans.Messaging[100071],ConnectionFailedto 9.90Orleans.Messaging[100071],HostUnreachableto 9.90Polly[3]Orleans.Placement/(null)/Retry, same connect failureOrleans.Messaging[100071], placementTimeoutException, innermost framePlacementService.cs:602(the directory-lookup await)Corrections to the issue texts:
fail: Polly[3] … Handled: True, Attempt: 3means the retries ran out. Polly logs the final handled attempt at Error, and Orleans'PlacementMaxRetries= 3. The message was rejected; the retry did not absorb it.Doc/Architecture/PeerDeathIsDatedByItsExceptionClass.What MeshWeaver does with it.
RoutingGrain.DeliverToGrainObservablere-sends a connection-failed rejection up to 6 times (about 10 s), then NACKs it asShuttingDown(IsDepartedSiloRejection). A placementTimeoutExceptionis never re-sent (IsResponseTimeout) and is NACKed as terminalFailed. Whether that is right is an open design question, written up in PR #5450.Left open (sev:L), blocked on one read that I could not take because the control instance answered 503. Graceful or crash? A graceful stop writes
ShuttingDownto the (Postgres) membership table before exit, yet the survivors kept addressing 9.90 for ≥ 36 s. That fits a row leftActiveuntil the survivors' probes voted it out, which takes about 75 s under Memex.Portal.Distributed's deliberateProbeTimeout15 s ×NumMissedProbesLimit5. It would equally fit a graceful stop whose table write failed: the same pods logged Postgres TLS-handshake cancellations at 14:42 and 14:44 (#5314). The act: aLogsaction onDeployments/memex-cloud, filtered to the5c444645f8pod that owned10.244.9.90(started 14:09:28Z), withquery: "Graceful shutdown aborted|Application is shutting down|NT_SIGINFO|Dump successfully written", plus that container's exit reason fromSample. If it crashed, the crash is the defect. If the stop was graceful and the table write failed, the Postgres path is. If the stop was graceful and the write succeeded, gossip propagation is.systemorph-com commentedon Sep 23, 2026
Still occurring: 14 more since the last update (193 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22d,memex-portal-deployment-56fbdcd48f-4dpcnRecent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 23, 2026
Still occurring: 13 more since the last update (206 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22d,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-7d7f4d84f9-mvzb5Recent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 26, 2026
Still occurring: 47 more since the last update (253 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22d,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-7d7f4d84f9-mvzb5,memex-portal-deployment-745745db7c-b2gjnRecent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 28, 2026
Still occurring: 20 more since the last update (273 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22d,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-7d7f4d84f9-mvzb5,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-8d85b9854-s66rpRecent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Sep 29, 2026
Still occurring: 15 more since the last update (288 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22d,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-7d7f4d84f9-mvzb5,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-8d85b9854-s66rp,memex-portal-deployment-8d85b9854-gtckfRecent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 1, 2026
Still occurring: 1 more since the last update (289 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22d,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-7d7f4d84f9-mvzb5,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-8d85b9854-s66rp,memex-portal-deployment-8d85b9854-gtckf,memex-portal-deployment-57b7d56fdd-5wqxqRecent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 1, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
08481dbd6614b2ac(Orleans.Messaging), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.f13960a22390eb3a08481dbd6614b2acOrleans.MessagingOrleans.Runtime.OrleansMessageRejectionExceptionException while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S1{n}.{n}:{n}:{n}. See InnerException ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to {n}.{n}:{n}. Error: ConnectionRefusedmemex-cloudmemex-portal-deployment-57b7d56fdd-5wqxqAdmin/_LogIncident/f13960a22390eb3aWhat triage wrote about this fingerprint
What is failing
The portal pod (
memex-portal-deployment-57b7d56fdd-5wqxq) cannot deliver a message-hub request (messagehub/Admin/UpdatePolicy, viaMeshWeaver.Connection→Orleans MessageCenter): the grain directory hands back siloS10.244.17.53:11111:149853740, a TCP connect to that endpoint is refused, and the message is rejected. The destination silo is gone but its address is still being used for placement.Probable cause — medium confidence
A stale silo generation in the Orleans cluster: the target pod has terminated (or been replaced) without being evicted from membership/directory in time, so
MessageCenter.SendMessage→ConnectionManager.ConnectAsynckeeps dialing a closed port. TheConnectionRefused(not timeout) inner error means nothing is listening at that address — the process is genuinely gone. The Polly pipeline (Orleans.Placement/(null)/Retry) swallowed the failure as Handled after its retry attempts, so delivery silently failed. The sibling incident59da89765aae6501is the same burst — same timestamps, pod, target endpoint and stack — captured at thePollylog site instead ofOrleans.Messaging; treat the two as one defect.This is also the same defect family already ticketed as #2307 (incident
f367c5512327cc57): stale silo generations keep receiving message-hub traffic after pod rollouts. That earlier episode ran for days across every replicaset because membership could not converge (clustering-table/Postgres unreachable, see incidents3d646146cf9677eaand222f37903cc8db90cited there). What this incident adds is theConnectionRefusedshape specifically — the old ticket's samples showwill retry after Nmsrejections; these show a hard refusal with no retry, i.e. the endpoint is not merely late, it is dead.Confidence is medium, not higher, because the evidence cannot distinguish the two flavors: (a) a normal, self-healing eviction race at rollout — the directory briefly serves a dead address until membership kicks it (in which case these 2 lines in 2.5s are ordinary churn and only the log noise is worth fixing), or (b) the persistent non-convergence mode of #2307 recurring. Occurrences are too few and too recent to tell. The first thing to check is whether
S10.244.17.53was a pod that rolled at ~12:08 UTC on 2026-10-01, and whether the rejection rate climbed or stopped.Impact
Minimal as logged: 2 occurrences on 1 pod within 2.5 seconds, and the target is a single admin message (
UpdatePolicy). The Polly retry already ran (attempt 3) before surfacing, so no user-visible path is demonstrably broken by this burst alone. Rated sev:L on these numbers — but if the recurrence pattern of #2307 resumes (continuous firing across pods), re-rule to sev:H, since message-hub delivery to real users was lost for days in that episode.Where to look
memex-portal-deploymentaround 2026-10-01 12:08 UTC — confirm a pod restart/rollout matches10.244.17.53and whether the rejections stopped once membership converged.MembershipTableManager,MembershipAgentlogs at the same time) — the known root cause from Mesh hub delivery retries departed silos in a ~1 Hz loop instead of re-resolving the grain #2307 was the clustering table (Postgres) being unreachable, which blocks both graceful shutdown writes and eviction.MeshWeaver.Connection.OrleansaroundDeliverMessage/IMessageHubGrain— a rejection on a dead silo generation should invalidate the cached directory entry and re-place rather than surface as a failed delivery; same containment item as in Mesh hub delivery retries departed silos in a ~1 Hz loop instead of re-resolving the grain #2307.Still occurring: 2 more since the last update (2 total).
memex-portal-deployment-57b7d56fdd-5wqxqRecent log lines
systemorph-com commentedon Oct 2, 2026
Still occurring: 3 more since the last update (292 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22d,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-7d7f4d84f9-mvzb5,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-8d85b9854-s66rp,memex-portal-deployment-8d85b9854-gtckf,memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-6f8489747-fchl7,memex-portal-deployment-6f8489747-49hwvRecent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 7, 2026
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-748f7f4577-d92xfRecent log lines
19 remaining items
systemorph-com commentedon Oct 18, 2026
Still occurring: 118 more since the last update (1523 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22d,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-7d7f4d84f9-mvzb5,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-8d85b9854-s66rp,memex-portal-deployment-8d85b9854-gtckf,memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-6f8489747-fchl7,memex-portal-deployment-6f8489747-49hwv,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-748f7f4577-pjrjp,memex-portal-deployment-748f7f4577-tp7b6,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2ww,memex-portal-deployment-7868c55d6-hfnt8Recent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 18, 2026
Still occurring: 6 more since the last update (40 total).
memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2wwRecent log lines
systemorph-com commentedon Oct 20, 2026
Still occurring: 5 more since the last update (1528 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22d,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-7d7f4d84f9-mvzb5,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-8d85b9854-s66rp,memex-portal-deployment-8d85b9854-gtckf,memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-6f8489747-fchl7,memex-portal-deployment-6f8489747-49hwv,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-748f7f4577-pjrjp,memex-portal-deployment-748f7f4577-tp7b6,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2ww,memex-portal-deployment-7868c55d6-hfnt8,memex-portal-deployment-85f94799c4-lw5sv,memex-portal-deployment-85f94799c4-5pjlrRecent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 20, 2026
Still occurring: 18 more since the last update (58 total).
memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2ww,memex-portal-deployment-85f94799c4-5pjlr,memex-portal-deployment-85f94799c4-lw5svRecent log lines
systemorph-com commentedon Oct 20, 2026
Still occurring: 23 more since the last update (63 total).
memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2ww,memex-portal-deployment-85f94799c4-5pjlr,memex-portal-deployment-85f94799c4-lw5svRecent log lines
systemorph-com commentedon Oct 20, 2026
Still occurring: 9 more since the last update (49 total).
memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2ww,memex-portal-deployment-85f94799c4-5pjlr,memex-portal-deployment-85f94799c4-lw5svRecent log lines
systemorph-com commentedon Oct 20, 2026
Still occurring: 14 more since the last update (63 total).
memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2ww,memex-portal-deployment-85f94799c4-5pjlr,memex-portal-deployment-85f94799c4-lw5svRecent log lines
systemorph-com commentedon Oct 20, 2026
Still occurring: 157 more since the last update (1685 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22d,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-7d7f4d84f9-mvzb5,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-8d85b9854-s66rp,memex-portal-deployment-8d85b9854-gtckf,memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-6f8489747-fchl7,memex-portal-deployment-6f8489747-49hwv,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-748f7f4577-pjrjp,memex-portal-deployment-748f7f4577-tp7b6,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2ww,memex-portal-deployment-7868c55d6-hfnt8,memex-portal-deployment-85f94799c4-lw5sv,memex-portal-deployment-85f94799c4-5pjlr,memex-portal-deployment-5fd6d747d-xbct6Recent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 22, 2026
Still occurring: 37 more since the last update (1722 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22d,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-7d7f4d84f9-mvzb5,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-8d85b9854-s66rp,memex-portal-deployment-8d85b9854-gtckf,memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-6f8489747-fchl7,memex-portal-deployment-6f8489747-49hwv,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-748f7f4577-pjrjp,memex-portal-deployment-748f7f4577-tp7b6,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2ww,memex-portal-deployment-7868c55d6-hfnt8,memex-portal-deployment-85f94799c4-lw5sv,memex-portal-deployment-85f94799c4-5pjlr,memex-portal-deployment-5fd6d747d-xbct6,memex-portal-deployment-78986f8447-m2gsv,memex-portal-deployment-78986f8447-2sgflRecent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 22, 2026
Still occurring: 9 more since the last update (72 total).
memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2ww,memex-portal-deployment-85f94799c4-5pjlr,memex-portal-deployment-85f94799c4-lw5sv,memex-portal-deployment-78986f8447-m2gsv,memex-portal-deployment-78986f8447-2sgflRecent log lines
systemorph-com commentedon Oct 23, 2026
Still occurring: 93 more since the last update (1815 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22d,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-7d7f4d84f9-mvzb5,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-8d85b9854-s66rp,memex-portal-deployment-8d85b9854-gtckf,memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-6f8489747-fchl7,memex-portal-deployment-6f8489747-49hwv,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-748f7f4577-pjrjp,memex-portal-deployment-748f7f4577-tp7b6,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2ww,memex-portal-deployment-7868c55d6-hfnt8,memex-portal-deployment-85f94799c4-lw5sv,memex-portal-deployment-85f94799c4-5pjlr,memex-portal-deployment-5fd6d747d-xbct6,memex-portal-deployment-78986f8447-m2gsv,memex-portal-deployment-78986f8447-2sgfl,memex-portal-deployment-78fbdddf6c-drjx8,memex-portal-deployment-57c8f675f9-z4bnjRecent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 23, 2026
Still occurring: 2 more since the last update (74 total).
memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2ww,memex-portal-deployment-85f94799c4-5pjlr,memex-portal-deployment-85f94799c4-lw5sv,memex-portal-deployment-78986f8447-m2gsv,memex-portal-deployment-78986f8447-2sgfl,memex-portal-deployment-57c8f675f9-z4bnjRecent log lines
systemorph-com commentedon Oct 23, 2026
Still occurring: 269 more since the last update (2084 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-89868db67-xg22d,memex-portal-deployment-56fbdcd48f-4dpcn,memex-portal-deployment-7d7f4d84f9-mvzb5,memex-portal-deployment-745745db7c-b2gjn,memex-portal-deployment-8d85b9854-s66rp,memex-portal-deployment-8d85b9854-gtckf,memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-6f8489747-fchl7,memex-portal-deployment-6f8489747-49hwv,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-748f7f4577-pjrjp,memex-portal-deployment-748f7f4577-tp7b6,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2ww,memex-portal-deployment-7868c55d6-hfnt8,memex-portal-deployment-85f94799c4-lw5sv,memex-portal-deployment-85f94799c4-5pjlr,memex-portal-deployment-5fd6d747d-xbct6,memex-portal-deployment-78986f8447-m2gsv,memex-portal-deployment-78986f8447-2sgfl,memex-portal-deployment-78fbdddf6c-drjx8,memex-portal-deployment-57c8f675f9-z4bnj,memex-portal-deployment-7bcd666f5-bzc8n,memex-portal-deployment-7bcd666f5-vpv79Recent log lines
Re-addressed by the current identity function: this incident inherited
f367c5512327cc57(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 23, 2026
Still occurring: 13 more since the last update (87 total).
memex-portal-deployment-57b7d56fdd-5wqxq,memex-portal-deployment-748f7f4577-d92xf,memex-portal-deployment-76cddc5847-x9fff,memex-portal-deployment-76cddc5847-f4gk9,memex-portal-deployment-5779f6fc79-kf2hr,memex-portal-deployment-5779f6fc79-g7ckx,memex-portal-deployment-d677f4c4d-cskdv,memex-portal-deployment-d677f4c4d-5x827,memex-portal-deployment-7868c55d6-z27mm,memex-portal-deployment-7868c55d6-jbp2d,memex-portal-deployment-677bd84df8-4s2ww,memex-portal-deployment-85f94799c4-5pjlr,memex-portal-deployment-85f94799c4-lw5sv,memex-portal-deployment-78986f8447-m2gsv,memex-portal-deployment-78986f8447-2sgfl,memex-portal-deployment-57c8f675f9-z4bnj,memex-portal-deployment-7bcd666f5-bzc8n,memex-portal-deployment-7bcd666f5-vpv79Recent log lines
rbuergi commentedon Oct 24, 2026
#7003 (with #7002 as its duplicate) is the end of the same 2026-10-20 14:15Z departure as this issue's newest samples. It shows the same departure at the moment membership declares the silo dead (
SiloUnavailableExceptionon the placement's directory lookup), after this issue'sConnectionRefusedattempts.ShuttingDown.Neither changes this issue's own shape. The recurrences posted here since 2026-10-17 are the connection-refused wording, which the rule already covers inside a roll window. I have not established why they were ticketed anyway. The likely explanation is that no shutdown line of the workload preceded them in the 5-minute window, i.e. departures that were kills or evictions; that is a reading still to be taken.