Grain directory deregistration fails during silo churn: batched unregister requests dropped with ConnectionFailedException, leaving stale directory entries #5730

Activity

1. systemorph-com

   added

   bugSomething isn't workingSomething isn't working

   sev:MMedium - secondary path broken, or easy workaroundMedium - secondary path broken, or easy workaround

   on Sep 26, 2026
2. rbuergi commented on Sep 26, 2026

   

   rbuergi

   on Sep 26, 2026

   Contributor

   More actions

   Closing as a duplicate of #5704, which is now the canonical issue for the silo-churn log sites. The evidence is in this comment.

   Summary. These lines come from Orleans' own DhtGrainLocator.BatchedDeregistrationWorker. It cannot reach the directory partition owner because that silo has gone. When membership declares the silo dead, Orleans runs directory repair, and no MeshWeaver code is on this path. The last sample (2026-09-24 13:43Z) falls in the same minute as a memex pod departure. The #5704 dequeue timeouts for 13:41–13:43Z show it, and their later target generation 149262209 is the boot of 56fbdcd48f-cbms9 at 13:43:29Z that replaced the silo.

   What still needs fixing is why memex pods keep leaving abruptly. That is tracked on #5704.
3. rbuergi

   closed this as not plannedon Sep 26, 2026
4. systemorph-com commented on Oct 17, 2026

   

   systemorph-comBot

   on Oct 17, 2026

   ContributorAuthor

   More actions

   Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.

   Still occurring: 40 more since the last update (639 total).

   • Last seen: 2026-10-17 14:48:58Z
   • Pods: memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-65fbcf6cd5-spgc9, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-7d6b864b54-w4xnl, memex-portal-deployment-85859db964-q5zcm, memex-portal-deployment-85859db964-7sqbk, memex-portal-deployment-5779f6fc79-kf2hr

   Recent log lines

   2026-10-17 14:48:58Z memex-portal-deployment-5779f6fc79-kf2hr fail: Orleans.Grain[1528537479]
         Failed to unregister activation '[Activation: S10.244.6.75:11111:151252394/messagehub/Chess/_Issue/1785@862ccd98c02241c3b04787fe2ac8fbcd#GrainType=MeshWeaver.Hosting.Orleans.MessageHubGrain,MeshWeaver.Hosting.Orleans Placement=PreferLocalPlacement State=Deactivating]' from directory
         Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.3.220:11111:151252295, will retry after 428.0117ms
            at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 142
            at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 245
            at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 100
            at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
         --- End of stack trace from previous location ---
            at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterManyAsync(List`1 addresses, UnregistrationCause cause, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 792
            at Orleans.Runtime.GrainDirectory.DhtGrainLocator.BatchedDeregistrationWorker.ProcessDeregistrationQueue() in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 141
            at Orleans.Runtime.ActivationData.FinishDeactivating(Deactivate deactivateCommand, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Catalog/ActivationData.cs:line 2143
   2026-10-17 14:48:58Z memex-portal-deployment-5779f6fc79-kf2hr fail: Orleans.Grain[1528537479]
         Failed to unregister activation '[Activation: S10.244.6.75:11111:151252394/messagehub/Planning/_Issue/201@254c237fdca94f19a15a7ea633e27b9c#GrainType=MeshWeaver.Hosting.Orleans.MessageHubGrain,MeshWeaver.Hosting.Orleans Placement=PreferLocalPlacement State=Deactivating]' from directory
         Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.3.220:11111:151252295, will retry after 426.9852ms
            at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 142
            at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 245
            at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 100
            at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
         --- End of stack trace from previous location ---
            at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterManyAsync(List`1 addresses, UnregistrationCause cause, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 792
            at Orleans.Runtime.GrainDirectory.DhtGrainLocator.BatchedDeregistrationWorker.ProcessDeregistrationQueue() in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 141
            at Orleans.Runtime.ActivationData.FinishDeactivating(Deactivate deactivateCommand, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Catalog/ActivationData.cs:line 2143
   2026-10-17 14:48:58Z memex-portal-deployment-5779f6fc79-kf2hr fail: Orleans.Grain[1528537479]
         Failed to unregister activation '[Activation: S10.244.6.75:11111:151252394/messagehub/rbuergi/MeshWeaverInstance/ci-education-r5@3aecc65c8d1941608fb89626782de193#GrainType=MeshWeaver.Hosting.Orleans.MessageHubGrain,MeshWeaver.Hosting.Orleans Placement=PreferLocalPlacement State=Deactivating]' from directory
         Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.3.220:11111:151252295, will retry after 423.0891ms
            at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 142
            at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 245
            at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 100
            at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
         --- End of stack trace from previous location ---
            at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterManyAsync(List`1 addresses, UnregistrationCause cause, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 792
            at Orleans.Runtime.GrainDirectory.DhtGrainLocator.BatchedDeregistrationWorker.ProcessDeregistrationQueue() in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 141
            at Orleans.Runtime.ActivationData.FinishDeactivating(Deactivate deactivateCommand, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Catalog/ActivationData.cs:line 2143
   

   _{Re-addressed by the current identity function: this incident inherited 2515e817b336ca61 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}
5. systemorph-com commented on Oct 20, 2026

   

   systemorph-comBot

   on Oct 20, 2026

   ContributorAuthor

   More actions

   Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.

   Still occurring: 746 more since the last update (1385 total).

   • Last seen: 2026-10-20 14:15:32Z
   • Pods: memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-65fbcf6cd5-spgc9, memex-portal-deployment-65fbcf6cd5-9t64t, memex-portal-deployment-7d6b864b54-w4xnl, memex-portal-deployment-85859db964-q5zcm, memex-portal-deployment-85859db964-7sqbk, memex-portal-deployment-5779f6fc79-kf2hr, memex-portal-deployment-85f94799c4-lw5sv

   Recent log lines

   2026-10-20 14:15:32Z memex-portal-deployment-85f94799c4-lw5sv fail: Orleans.Grain[1528537479]
         Failed to unregister activation '[Activation: S10.244.5.170:11111:151508593/messagehub/Parties/_Entitlements/pierre35@c8c39dc564db4901bed5f072dbc482d3#GrainType=MeshWeaver.Hosting.Orleans.MessageHubGrain,MeshWeaver.Hosting.Orleans Placement=PreferLocalPlacement State=Deactivating]' from directory
         Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.3.115:11111:151508771, will retry after 971.1428ms
            at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 142
            at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 245
            at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 100
            at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
         --- End of stack trace from previous location ---
            at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterManyAsync(List`1 addresses, UnregistrationCause cause, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 792
            at Orleans.Runtime.GrainDirectory.DhtGrainLocator.BatchedDeregistrationWorker.ProcessDeregistrationQueue() in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 141
            at Orleans.Runtime.ActivationData.FinishDeactivating(Deactivate deactivateCommand, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Catalog/ActivationData.cs:line 2143
   2026-10-20 14:15:32Z memex-portal-deployment-85f94799c4-lw5sv fail: Orleans.Grain[1528537479]
         Failed to unregister activation '[Activation: S10.244.5.170:11111:151508593/messagehub/rbuergi/MeshWeaverInstance/ci-reinsurance-r5@502f8ffb69594bc6bbb157aa4d90f586#GrainType=MeshWeaver.Hosting.Orleans.MessageHubGrain,MeshWeaver.Hosting.Orleans Placement=PreferLocalPlacement State=Deactivating]' from directory
         Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.3.115:11111:151508771, will retry after 971.1428ms
            at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 142
            at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 245
            at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 100
            at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
         --- End of stack trace from previous location ---
            at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterManyAsync(List`1 addresses, UnregistrationCause cause, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 792
            at Orleans.Runtime.GrainDirectory.DhtGrainLocator.BatchedDeregistrationWorker.ProcessDeregistrationQueue() in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 141
            at Orleans.Runtime.ActivationData.FinishDeactivating(Deactivate deactivateCommand, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Catalog/ActivationData.cs:line 2143
   2026-10-20 14:15:32Z memex-portal-deployment-85f94799c4-lw5sv fail: Orleans.Grain[1528537479]
         Failed to unregister activation '[Activation: S10.244.5.170:11111:151508593/messagehub/Training/_Entitlements/luca@184abf7eb5d04e57b605735d3bfb55e0#GrainType=MeshWeaver.Hosting.Orleans.MessageHubGrain,MeshWeaver.Hosting.Orleans Placement=PreferLocalPlacement State=Deactivating]' from directory
         Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.3.115:11111:151508771, will retry after 971.1428ms
            at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 142
            at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 245
            at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 100
            at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
         --- End of stack trace from previous location ---
            at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterManyAsync(List`1 addresses, UnregistrationCause cause, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 792
            at Orleans.Runtime.GrainDirectory.DhtGrainLocator.BatchedDeregistrationWorker.ProcessDeregistrationQueue() in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 141
            at Orleans.Runtime.ActivationData.FinishDeactivating(Deactivate deactivateCommand, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Catalog/ActivationData.cs:line 2143
   

   _{Re-addressed by the current identity function: this incident inherited 2515e817b336ca61 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}
6. systemorph-com commented on Oct 20, 2026

   

   systemorph-comBot

   on Oct 20, 2026

   ContributorAuthor

   More actions

   Folded in — a DIFFERENT fingerprint from the same log site.

   This issue tracks the log site aaf15f61b2ed8836 (Orleans.Grain), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.

   Folded fingerprint	257c678c4bd580d7
   Log site	aaf15f61b2ed8836
   Category	Orleans.Grain
   Severity	Error
   Exception	Orleans.Runtime.OrleansMessageRejectionException
   Detail	Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S1{n}.{n}:{n}:{n}. See InnerException ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to {n}.{n}:{n}. Error: ConnectionRefused
   Namespace	memex-cloud
   Pods	memex-portal-deployment-85f94799c4-lw5sv
   Occurrences	804
   First seen	2026-10-20 14:15:30Z
   Last seen	2026-10-20 14:15:32Z
   Already folded here	1 fingerprint(s)
   Incident	Admin/_LogIncident/257c678c4bd580d7

   What triage wrote about this fingerprint

   Observed

   MeshWeaver.Hosting.Orleans.MessageHubGrain activations, while deactivating on silo S10.244.5.170:11111:151508593 (pod memex-portal-deployment-85f94799c4-lw5sv), failed to unregister from the grain directory. Each unregister message was rejected with Orleans.Runtime.OrleansMessageRejectionException wrapping ConnectionFailedException / SocketConnectionException: ConnectionRefused — the peer silo S10.244.3.115:11111:151508771 refused the TCP connection (Orleans SocketConnectionFactory.ConnectAsync → ConnectionManager.ConnectAsync → MessageCenter.SendMessage). 804 distinct MessageHubGrain activations across many tenants were affected (samples show messagehub/{Portal}/_Entitlements/{user} grains for HomeAssistant, Parties, AppleWeather, AppleMaps, Northwind, RemoteControl, DoublePendulum, and a messagehub/AppleWeather/_Policy grain).

   Where and when

   All 804 occurrences fall in one burst of ~2.5 seconds (2026-10-20 14:15:30–14:15:32 UTC) on a single pod. The failing silo and the unreachable silo are different pods (10.244.5.170 vs 10.244.3.115). Every sample in the burst carries the same target silo — nothing else failed.

   Impact

   Short-lived, single-pod, shutdown-time window. Directory entries for the deactivated activations may remain stale in the directory partition owned by the unreachable silo until cleaned by other means; no user-visible failure is evidenced by this incident alone. Not established whether any client call actually failed.

   Not established

   • Whether this pod or the target pod 10.244.3.115 was terminating at the time (a rollout, a scale-down, a crash) — this burst is consistent with all of these.
   • Whether 804 deactivations in 2.5s means the pod itself was shutting down (draining all activations), or a placement/keepalive change deactivated them.
   • Whether the directory eventually converged (stale registrations cleaned up) — not visible in these log lines.
   • Whether other pods logged the mirrored failure against this pod.

   Candidate causes and the reading that separates them

   1. Rolling-restart shutdown race — this pod was terminating and 10.244.3.115 (another portal pod) had already stopped, so the directory unregisters failed. Predicts: Kubernetes pod-termination events for both pods at ~14:15 UTC and a deployment rollout in progress at that time. Reading: the deployment's rollout history and pod Terminating timestamps for the IPs above.
   2. Ungraceful death of the directory-owning silo — 10.244.3.115 crashed or was OOM-killed. Predicts: a non-zero exit or an Killed event for that pod, and possibly other incident fingerprints (membership-gossip warnings, connection-drop logs on the surviving pods). Reading: pod events and silo-membership log lines around 14:15.
   3. Stale directory placement — the directory routing pointed at a silo address that no longer exists at all (e.g. after a re-deploy changed pod IPs). Predicts: the target silo was gone long before the burst and similar rejections would appear outside the shutdown window too. Reading: whether S10.244.3.115:11111:151508771 appears in the cluster membership table at that time.

   Where to look

   • MeshWeaver.Hosting.Orleans.MessageHubGrain (assembly MeshWeaver.Hosting.Orleans) — the grain type whose deactivations failed to unregister.
   • Orleans Catalog/DirectoryService.UnregisterAsync path and Orleans.Runtime.Messaging.MessageCenter.SendMessage / ConnectionManager.GetConnectionAsync — where the unregister message was rejected (framework stack in the samples).
   • Kubernetes events for pods memex-portal-deployment-85f94799c4-lw5sv and the pod holding 10.244.3.115, around 2026-10-20 14:15 UTC.

   Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.

   Still occurring: 804 more since the last update (804 total).

   • Last seen: 2026-10-20 14:15:32Z
   • Pods: memex-portal-deployment-85f94799c4-lw5sv

   Recent log lines

   2026-10-20 14:15:32Z memex-portal-deployment-85f94799c4-lw5sv fail: Orleans.Grain[1528537479]
         Failed to unregister activation '[Activation: S10.244.5.170:11111:151508593/messagehub/AppleWeather/_Policy@640ee29d946c458a8fe29c7fb4408958#GrainType=MeshWeaver.Hosting.Orleans.MessageHubGrain,MeshWeaver.Hosting.Orleans Placement=PreferLocalPlacement State=Deactivating]' from directory
         Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.244.3.115:11111:151508771. See InnerException
          ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.244.3.115:11111. Error: ConnectionRefused
            at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 67
            at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/ConnectionFactory.cs:line 66
            at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 257
            --- End of inner exception stack trace ---
            at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 296
            at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 149
            at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 245
            at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) i…[truncated]
   2026-10-20 14:15:32Z memex-portal-deployment-85f94799c4-lw5sv fail: Orleans.Grain[1528537479]
         Failed to unregister activation '[Activation: S10.244.5.170:11111:151508593/messagehub/RemoteControl/_Entitlements/rbuergi@231c26472fb84f44b4c62042db244d34#GrainType=MeshWeaver.Hosting.Orleans.MessageHubGrain,MeshWeaver.Hosting.Orleans Placement=PreferLocalPlacement State=Deactivating]' from directory
         Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.244.3.115:11111:151508771. See InnerException
          ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.244.3.115:11111. Error: ConnectionRefused
            at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 67
            at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/ConnectionFactory.cs:line 66
            at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 257
            --- End of inner exception stack trace ---
            at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 296
            at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 149
            at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 245
            at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult…[truncated]
   2026-10-20 14:15:32Z memex-portal-deployment-85f94799c4-lw5sv fail: Orleans.Grain[1528537479]
         Failed to unregister activation '[Activation: S10.244.5.170:11111:151508593/messagehub/DoublePendulum/_Entitlements/martinrolandwuensche@36ac31b0cffe4eeab9d40b204fa0dfee#GrainType=MeshWeaver.Hosting.Orleans.MessageHubGrain,MeshWeaver.Hosting.Orleans Placement=PreferLocalPlacement State=Deactivating]' from directory
         Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.244.3.115:11111:151508771. See InnerException
          ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.244.3.115:11111. Error: ConnectionRefused
            at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 67
            at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/ConnectionFactory.cs:line 66
            at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 257
            --- End of inner exception stack trace ---
            at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 296
            at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 149
            at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|32_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 245
            at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSo…[truncated]