Skip to content

Orleans grain placement for messagehub grains times out, dropping a DeliverMessage call #5334

Closed as duplicate
Closed as duplicate
Orleans grain placement for messagehub grains times out, dropping a DeliverMessage call#5334
Labels
bugSomething isn't workingsev:LLow - cosmetic, rare edge case, or developer-only

Description

@systemorph-com
systemorph-com
bot
opened on Sep 22, 2026

What is failing

A single DeliverMessage call to the message-hub grain for Doc/Architecture/PolicyNotProse failed to be addressed: Orleans' grain placement for messagehub/Doc/Architecture/PolicyNotProse did not complete within the 30 s Polly timeout (PlacementService.PlacementWorker.ExecutePlacementAsync), so the request was rejected before it ever reached the grain. Two occurrences, same timestamp, one portal pod.

Probable cause (low-moderate confidence)

The stack is entirely Orleans runtime placement machinery (no MeshWeaver frame beyond the message envelope). A 30 s placement timeout almost always means the placement worker queue was starved or blocked — consistent with the cluster-wide burst of unrelated TimeoutException incidents (memory queue reads, DataContext initialization) seen on the same memex-cloud deployment in the surrounding days. I could not find a code defect in the message-hub path itself; the evidence points at transient runtime/cluster overload, not a MeshWeaver bug. If this recurs, capture placement-worker queue depth and silo health at the time.

Impact

Minimal as observed: 2 failed deliveries of one message to one node's hub, single pod, no recurrence in ~5 h since. Potentially a dropped update/subscription delivery for that one node's clients at that moment.

Where to look

  • Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync (src/Orleans.Runtime/Placement/PlacementService.cs:602) — why the placement task never completed within 30 s.
  • The sibling incidents of the same period (Admin/_LogIncident/17309c2fb8611d33 memory-0 timeouts, 1eb776bfdfebfc70 DataContext init timeout) suggest a shared overload window; compare timelines before code-diving.
  • Related but distinct: MessageHubGrain activation frequently times out during initialization due to unresolvable mesh nodes #1186 covers MessageHubGrain activation timeouts from unclaimed partitions — different symptom, not a duplicate.

Evidence
Fingerprint f7d8f9982b20cce5
Category Orleans.Messaging
Severity Error
Exception System.TimeoutException
Top frame Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
Namespace memex-cloud
Pods memex-portal-deployment-89868db67-xg22d
Occurrences 2
First seen 2026-09-22 14:41:30Z
Last seen 2026-09-22 14:41:30Z
Routing not determined — no configured route matches the category Orleans.Messaging. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-09-22 14:41:30Z memex-portal-deployment-89868db67-xg22d fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.195:11111:149092846 sys.client/hosted-10.244.2.195:11111@149092846]->[ messagehub/Doc/Architecture/PolicyNotProse] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #2050A52DBDF20FC5
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Doc/Architecture/PolicyNotProse.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunConti…[truncated]
2026-09-22 14:41:30Z memex-portal-deployment-89868db67-xg22d fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.195:11111:149092846 sys.client/hosted-10.244.2.195:11111@149092846]->[ messagehub/Doc/Architecture/PolicyNotProse] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #2050A52DBDF20F05
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Doc/Architecture/PolicyNotProse.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunConti…[truncated]

Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

Opened automatically from Admin/_LogIncident/f7d8f9982b20cce5. Recurrences are folded into this issue rather than opening new ones.

Activity

  1. systemorph-com
    added
    bugSomething isn't working
    sev:LLow - cosmetic, rare edge case, or developer-only
    on Sep 22, 2026

  2. systemorph-com commented on Sep 22, 2026

    @systemorph-com
    systemorph-combot
    on Sep 22, 2026
    ContributorAuthor

    Still occurring: 6 more since the last update (8 total).

    • Last seen: 2026-09-22 14:41:38Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb
    Recent log lines 
    2026-09-22 14:41:34Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/jeandavid.goetz/_Install/DataModelling] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B4861DE
      System.TimeoutException: Grain placement operation timed out for grain messagehub/jeandavid.goetz/_Install/DataModelling.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext …[truncated]
2026-09-22 14:41:38Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/HomeAssistant/_Access/beat_Access] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B487AFB
      System.TimeoutException: Grain placement operation timed out for grain messagehub/HomeAssistant/_Access/beat_Access.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionC…[truncated]
2026-09-22 14:41:38Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/HomeAssistant/_Access/beat_Access] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B487E6F
      System.TimeoutException: Grain placement operation timed out for grain messagehub/HomeAssistant/_Access/beat_Access.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionC…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  3. systemorph-com commented on Sep 22, 2026

    @systemorph-com
    systemorph-combot
    on Sep 22, 2026
    ContributorAuthor

    Still occurring: 2 more since the last update (10 total).

    • Last seen: 2026-09-22 14:41:41Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb
    Recent log lines 
    2026-09-22 14:41:38Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/HomeAssistant/_Access/beat_Access] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B487E6F
      System.TimeoutException: Grain placement operation timed out for grain messagehub/HomeAssistant/_Access/beat_Access.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionC…[truncated]
2026-09-22 14:41:41Z memex-portal-deployment-89868db67-xg22d fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.195:11111:149092846 sys.client/hosted-10.244.2.195:11111@149092846]->[ messagehub/rbuergi/MeshWeaverInstance/ci-socialmedia-r5] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #2050A52DBDF20F5A
      System.TimeoutException: Grain placement operation timed out for grain messagehub/rbuergi/MeshWeaverInstance/ci-socialmedia-r5.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Thr…[truncated]
2026-09-22 14:41:41Z memex-portal-deployment-89868db67-xg22d fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.195:11111:149092846 sys.client/hosted-10.244.2.195:11111@149092846]->[ messagehub/rbuergi/MeshWeaverInstance/ci-socialmedia-r5] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #2050A52DBDF20CBC
      System.TimeoutException: Grain placement operation timed out for grain messagehub/rbuergi/MeshWeaverInstance/ci-socialmedia-r5.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Thr…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  4. rbuergi commented on Sep 23, 2026

    @rbuergi
    rbuergi
    on Sep 23, 2026
    Contributor

    Duplicate of #5336, the one silo departure on memex-cloud at 2026-09-22 14:41Z.

    Grain placement operation timed out for messagehub/Doc/Architecture/PolicyNotProse on 89868db67-xg22d (plus 5c444645f8-8pdzb in the later updates), 14:41:30-14:41:41Z. The innermost frame, PlacementService.cs:602 in Orleans v10.3.1, is await lookup.AsTask().WaitAsync(ct), i.e. the directory lookup. Its 30 s budget therefore started at about 14:41:00, when the directory owner 9.90 had stopped answering, and ran out as the refusals to that silo began. This is the "mute, then gone" shape. It is not a local placement worker stalled by CPU starvation (#5037's reading).

    The shared site and discriminator: an Orleans directory lookup or placement into the partition of the departed silo S10.244.9.90:11111:149090968, during its ~36 s window, on the same two survivor pods, with no MeshWeaver frame. The grouping, what MeshWeaver does with the rejected DeliverMessage, and the one read still owed (graceful stop vs. crash) are on #5336 (#5336 (comment)) and in PR #5450.

  5. rbuergi
    closed this as duplicateon Sep 23, 2026

  6. systemorph-com commented on Sep 23, 2026

    @systemorph-com
    systemorph-combot
    on Sep 23, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 4 more since the last update (14 total).

    • Last seen: 2026-09-23 13:32:23Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2
    Recent log lines 
    2026-09-23 13:32:21Z memex-portal-deployment-5f95fb94dc-jvgv2 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.154:11111:149174495 sys.client/hosted-10.244.3.154:11111@149174495]->[ messagehub/Admin/_PluginGrant/ci-reinsurance-r5] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #5BBC1D346A597B06
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Admin/_PluginGrant/ci-reinsurance-r5.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext exec…[truncated]
2026-09-23 13:32:23Z memex-portal-deployment-5576ddf6f4-c5mz2 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.100:11111:149175078 sys.client/hosted-10.244.5.100:11111@149175078]->[ messagehub/Skill] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #5E6F2086F71DD47E
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Skill.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Syst…[truncated]
2026-09-23 13:32:23Z memex-portal-deployment-5576ddf6f4-c5mz2 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.100:11111:149175078 sys.client/hosted-10.244.5.100:11111@149175078]->[ messagehub/Skill] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #5E6F2086F71DD45B
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Skill.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Syst…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  7. systemorph-com commented on Sep 23, 2026

    @systemorph-com
    systemorph-combot
    on Sep 23, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 13 more since the last update (27 total).

    • Last seen: 2026-09-23 14:22:28Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn
    Recent log lines 
    2026-09-23 14:22:28Z memex-portal-deployment-777677694-p6hbn fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.120:11111:149177644 sys.client/hosted-10.244.5.120:11111@149177644]->[ messagehub/felice.buergi] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #53FC0B57D5A1DB74
      System.TimeoutException: Grain placement operation timed out for grain messagehub/felice.buergi.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object…[truncated]
2026-09-23 14:22:28Z memex-portal-deployment-777677694-p6hbn fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.120:11111:149177644 sys.client/hosted-10.244.5.120:11111@149177644]->[ messagehub/felice.buergi/AnimalMemory/_Activity/compile-state] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #53FC0B57D5A1DB76
      System.TimeoutException: Grain placement operation timed out for grain messagehub/felice.buergi/AnimalMemory/_Activity/compile-state.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunIn…[truncated]
2026-09-23 14:22:28Z memex-portal-deployment-777677694-p6hbn fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.120:11111:149177644 sys.client/hosted-10.244.5.120:11111@149177644]->[ messagehub/felice.buergi/AnimalMemory/_Activity/compile-state] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #53FC0B57D5A1C5A8
      System.TimeoutException: Grain placement operation timed out for grain messagehub/felice.buergi/AnimalMemory/_Activity/compile-state.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunIn…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  8. systemorph-com commented on Sep 23, 2026

    @systemorph-com
    systemorph-combot
    on Sep 23, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 8 more since the last update (35 total).

    • Last seen: 2026-09-23 17:58:38Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b
    Recent log lines 
    2026-09-23 17:58:22Z memex-portal-deployment-7d7f4d84f9-mvzb5 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.93:11111:149190807 sys.client/hosted-10.244.5.93:11111@149190807]->[ messagehub/MeshWeaverInstance/03349f5b76b5] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #5D5257E5CBE655D9
      System.TimeoutException: Grain placement operation timed out for grain messagehub/MeshWeaverInstance/03349f5b76b5.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinu…[truncated]
2026-09-23 17:58:38Z memex-portal-deployment-7d7f4d84f9-n7g6b fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.9.163:11111:149190240 sys.client/hosted-10.244.9.163:11111@149190240]->[ messagehub/Hosting/PlatformBuilds/plugins] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #2F39412CFB5DCDE5
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Hosting/PlatformBuilds/plugins.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext…[truncated]
2026-09-23 17:58:38Z memex-portal-deployment-7d7f4d84f9-n7g6b fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.9.163:11111:149190240 sys.client/hosted-10.244.9.163:11111@149190240]->[ messagehub/Hosting/PlatformBuilds/plugins] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #2F39412CFB5DC324
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Hosting/PlatformBuilds/plugins.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  9. systemorph-com commented on Sep 26, 2026

    @systemorph-com
    systemorph-combot
    on Sep 26, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 2 more since the last update (86 total).

    • Last seen: 2026-09-24 06:51:09Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp
    Recent log lines 
    2026-09-24 06:29:19Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.41:11111:149226787 sys.client/hosted-10.244.2.41:11111@149226787]->[ messagehub/Acp] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #3E90DCEC8F874EA5[ForwardCount=1]
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Acp.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state…[truncated]
2026-09-24 06:51:09Z memex-portal-deployment-76cf776847-v4txp fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.33:11111:149237088 sys.client/hosted-10.244.5.33:11111@149237088]->[ messagehub/Admin/Build] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #60099C7B1E760D53
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Admin/Build.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state…[truncated]
2026-09-24 06:51:09Z memex-portal-deployment-76cf776847-v4txp fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.33:11111:149237088 sys.client/hosted-10.244.5.33:11111@149237088]->[ messagehub/Admin/Build] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #60099C7B1E760D1F
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Admin/Build.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  10. systemorph-com commented on Sep 26, 2026

    @systemorph-com
    systemorph-combot
    on Sep 26, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 9 more since the last update (95 total).

    • Last seen: 2026-09-24 10:02:06Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-745745db7c-b2gjn
    Recent log lines 
    2026-09-24 10:02:06Z memex-portal-deployment-56fbdcd48f-sz5st fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.9.67:11111:149247132 sys.client/hosted-10.244.9.67:11111@149247132]->[ messagehub/Doc/Architecture] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #1AEC08E011662C26
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Doc/Architecture.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Ob…[truncated]
2026-09-24 10:02:06Z memex-portal-deployment-56fbdcd48f-sz5st fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.9.67:11111:149247132 sys.client/hosted-10.244.9.67:11111@149247132]->[ messagehub/Doc/Architecture] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #1AEC08E011662D51
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Doc/Architecture.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Ob…[truncated]
2026-09-24 10:02:06Z memex-portal-deployment-56fbdcd48f-sz5st fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.9.67:11111:149247132 sys.client/hosted-10.244.9.67:11111@149247132]->[ messagehub/Doc/Architecture] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #1AEC08E011662E6C
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Doc/Architecture.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Ob…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  11. systemorph-com commented on Sep 26, 2026

    @systemorph-com
    systemorph-combot
    on Sep 26, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 118 more since the last update (213 total).

    • Last seen: 2026-09-24 13:43:19Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-56fbdcd48f-7cd4h, memex-portal-deployment-56fbdcd48f-cbms9
    Recent log lines 
    2026-09-24 13:43:19Z memex-portal-deployment-745745db7c-b2gjn fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.80:11111:149257404 sys.client/hosted-10.244.5.80:11111@149257404]->[ messagehub/Hosting/TriageStatus/_Activity/compile-202609241342350479784a39d5d4943739e85bdfec78dfa19] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #36D24B49656FAC74
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Hosting/TriageStatus/_Activity/compile-202609241342350479784a39d5d4943739e85bdfec78dfa19.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext …[truncated]
2026-09-24 13:43:19Z memex-portal-deployment-745745db7c-b2gjn fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.80:11111:149257404 sys.client/hosted-10.244.5.80:11111@149257404]->[ messagehub/LinkedIn/Invitation/_Activity/compile-20260924134235573463ad4b3bd3744589c0e55b9210c75f4] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #36D24B49656FA399
      System.TimeoutException: Grain placement operation timed out for grain messagehub/LinkedIn/Invitation/_Activity/compile-20260924134235573463ad4b3bd3744589c0e55b9210c75f4.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext co…[truncated]
2026-09-24 13:43:19Z memex-portal-deployment-745745db7c-b2gjn fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.80:11111:149257404 sys.client/hosted-10.244.5.80:11111@149257404]->[ messagehub/LinkedIn/Invitation/_Activity/compile-20260924134235573463ad4b3bd3744589c0e55b9210c75f4] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #36D24B49656FAC64
      System.TimeoutException: Grain placement operation timed out for grain messagehub/LinkedIn/Invitation/_Activity/compile-20260924134235573463ad4b3bd3744589c0e55b9210c75f4.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext co…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  12. systemorph-com commented on Sep 26, 2026

    @systemorph-com
    systemorph-combot
    on Sep 26, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 11 more since the last update (224 total).

    • Last seen: 2026-09-25 15:28:44Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-56fbdcd48f-7cd4h, memex-portal-deployment-56fbdcd48f-cbms9
    Recent log lines 
    2026-09-25 15:28:42Z memex-portal-deployment-56fbdcd48f-cbms9 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.247:11111:149262209 sys.client/hosted-10.244.3.247:11111@149262209]->[ messagehub/rbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hub] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #2FE969D8E35A7B63
      System.TimeoutException: Grain placement operation timed out for grain messagehub/rbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hub.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at S…[truncated]
2026-09-25 15:28:42Z memex-portal-deployment-56fbdcd48f-cbms9 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.247:11111:149262209 sys.client/hosted-10.244.3.247:11111@149262209]->[ messagehub/rbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hub] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #2FE969D8E35A79B0
      System.TimeoutException: Grain placement operation timed out for grain messagehub/rbuergi/Feedback/20260922-per-request-registrants-never-leave-the-hub.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at S…[truncated]
2026-09-25 15:28:44Z memex-portal-deployment-56fbdcd48f-cbms9 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.247:11111:149262209 sys.client/hosted-10.244.3.247:11111@149262209]->[ podhub/portal/jFfcZid-c798bf25OYr2SrwKwwPyXzJui99qrM_ydoI] MeshWeaver.Connection.Orleans.IPodHubGrain.Attach() #2FE969D8E35A793E
      System.TimeoutException: Grain placement operation timed out for grain podhub/portal/jFfcZid-c798bf25OYr2SrwKwwPyXzJui99qrM_ydoI.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  13. systemorph-com commented on Sep 27, 2026

    @systemorph-com
    systemorph-combot
    on Sep 27, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 2 more since the last update (226 total).

    • Last seen: 2026-09-27 11:02:40Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-56fbdcd48f-7cd4h, memex-portal-deployment-56fbdcd48f-cbms9, memex-portal-deployment-555d9cd496-qntsh
    Recent log lines 
    2026-09-25 15:28:44Z memex-portal-deployment-56fbdcd48f-cbms9 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.247:11111:149262209 sys.client/hosted-10.244.3.247:11111@149262209]->[ podhub/portal/jFfcZid-c798bf25OYr2SrwKwwPyXzJui99qrM_ydoI] MeshWeaver.Connection.Orleans.IPodHubGrain.Attach() #2FE969D8E35A793E
      System.TimeoutException: Grain placement operation timed out for grain podhub/portal/jFfcZid-c798bf25OYr2SrwKwwPyXzJui99qrM_ydoI.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback…[truncated]
2026-09-27 11:02:40Z memex-portal-deployment-555d9cd496-qntsh fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.64:11111:149494762 sys.client/hosted-10.244.3.64:11111@149494762]->[ messagehub/Ops/Logs/memex-cloud-1790253968609807651-f8f895-cltgg] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #4C305F5442398C8C
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Ops/Logs/memex-cloud-1790253968609807651-f8f895-cltgg.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.R…[truncated]
2026-09-27 11:02:40Z memex-portal-deployment-555d9cd496-qntsh fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.64:11111:149494762 sys.client/hosted-10.244.3.64:11111@149494762]->[ messagehub/Ops/Logs/memex-cloud-1790253968609807651-f8f895-cltgg] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #4C305F5442398CA1
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Ops/Logs/memex-cloud-1790253968609807651-f8f895-cltgg.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.R…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  14. systemorph-com commented on Sep 27, 2026

    @systemorph-com
    systemorph-combot
    on Sep 27, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 12 more since the last update (238 total).

    • Last seen: 2026-09-27 11:02:45Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-56fbdcd48f-7cd4h, memex-portal-deployment-56fbdcd48f-cbms9, memex-portal-deployment-555d9cd496-qntsh
    Recent log lines 
    2026-09-27 11:02:45Z memex-portal-deployment-555d9cd496-qntsh fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.64:11111:149494762 sys.client/hosted-10.244.3.64:11111@149494762]->[ messagehub/Ops/Logs/memex-cloud-1790254003871494485-f8f895-ghgjt] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #4C305F5442398C94
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Ops/Logs/memex-cloud-1790254003871494485-f8f895-ghgjt.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.R…[truncated]
2026-09-27 11:02:45Z memex-portal-deployment-555d9cd496-qntsh fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.64:11111:149494762 sys.client/hosted-10.244.3.64:11111@149494762]->[ messagehub/Ops/Logs/memex-cloud-1790253968609485257-f8f895-cltgg] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #4C305F5442398C82
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Ops/Logs/memex-cloud-1790253968609485257-f8f895-cltgg.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.R…[truncated]
2026-09-27 11:02:45Z memex-portal-deployment-555d9cd496-qntsh fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.64:11111:149494762 sys.client/hosted-10.244.3.64:11111@149494762]->[ messagehub/Ops/Logs/memex-cloud-1790253968609485257-f8f895-cltgg] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #4C305F5442398C8B
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Ops/Logs/memex-cloud-1790253968609485257-f8f895-cltgg.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.R…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  15. 13 remaining items

  16. systemorph-com commented on Oct 11, 2026

    @systemorph-com
    systemorph-combot
    on Oct 11, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 36 more since the last update (684 total).

    • Last seen: 2026-10-11 03:58:05Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-56fbdcd48f-7cd4h, memex-portal-deployment-56fbdcd48f-cbms9, memex-portal-deployment-555d9cd496-qntsh, memex-portal-deployment-54bd7f5466-vnxfz, memex-portal-deployment-57b7d56fdd-5wqxq, memex-portal-deployment-58bf75c9d8-zmq48, memex-portal-deployment-76dd69c7fc-xxql9, memex-portal-deployment-76dd69c7fc-6tjhj, memex-portal-deployment-76dd69c7fc-xzsdl, memex-portal-deployment-6cfcf78898-lsklh, memex-portal-deployment-65f6466887-gv9qs, memex-portal-deployment-6cdc9bf66f-mbfsl, memex-portal-deployment-5f6b9fc9b-hrdhq, memex-portal-deployment-845898d964-mwzl7, memex-portal-deployment-845898d964-ptszc, memex-portal-deployment-845898d964-f6wm5, memex-portal-deployment-7648c9c56f-7nf5j, memex-portal-deployment-85587f8869-prgnp, memex-portal-deployment-85587f8869-z9bqf, memex-portal-deployment-5499f76b76-sz46x, memex-portal-deployment-55fffcbc56-4zd9g, memex-portal-deployment-5499f76b76-ltlwc, memex-portal-deployment-55fffcbc56-knljr, memex-portal-deployment-74bfbdff9-xmfpc, memex-portal-deployment-6749fbfc59-mz4k5, memex-portal-deployment-6749fbfc59-p29vl
    Recent log lines 
    2026-10-11 03:57:50Z memex-portal-deployment-74bfbdff9-xmfpc fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.6.197:11111:150693988 sys.client/hosted-10.244.6.197:11111@150693988]->[ messagehub/Admin/_Build/Systemorph.MeshWeaver.Plugins] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #1985E0E9D446E50A
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Admin/_Build/Systemorph.MeshWeaver.Plugins.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.Ex…[truncated]
2026-10-11 03:57:54Z memex-portal-deployment-74bfbdff9-xmfpc fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.6.197:11111:150693988 sys.client/hosted-10.244.6.197:11111@150693988]->[ messagehub/Ops/Issue/control__job-memex-migration-roll-3-1-10372__JobFailed] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #1985E0E9D446E692
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Ops/Issue/control__job-memex-migration-roll-3-1-10372__JobFailed.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
        …[truncated]
2026-10-11 03:58:05Z memex-portal-deployment-74bfbdff9-xmfpc fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.6.197:11111:150693988 sys.client/hosted-10.244.6.197:11111@150693988]->[ messagehub/AppleWeather/Weather/Test/WeatherTestsArea] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #1985E0E9D446FE86
      System.TimeoutException: Grain placement operation timed out for grain messagehub/AppleWeather/Weather/Test/WeatherTestsArea.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.Ex…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  17. systemorph-com commented on Oct 11, 2026

    @systemorph-com
    systemorph-combot
    on Oct 11, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 44 more since the last update (692 total).

    • Last seen: 2026-10-11 03:59:19Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-56fbdcd48f-7cd4h, memex-portal-deployment-56fbdcd48f-cbms9, memex-portal-deployment-555d9cd496-qntsh, memex-portal-deployment-54bd7f5466-vnxfz, memex-portal-deployment-57b7d56fdd-5wqxq, memex-portal-deployment-58bf75c9d8-zmq48, memex-portal-deployment-76dd69c7fc-xxql9, memex-portal-deployment-76dd69c7fc-6tjhj, memex-portal-deployment-76dd69c7fc-xzsdl, memex-portal-deployment-6cfcf78898-lsklh, memex-portal-deployment-65f6466887-gv9qs, memex-portal-deployment-6cdc9bf66f-mbfsl, memex-portal-deployment-5f6b9fc9b-hrdhq, memex-portal-deployment-845898d964-mwzl7, memex-portal-deployment-845898d964-ptszc, memex-portal-deployment-845898d964-f6wm5, memex-portal-deployment-7648c9c56f-7nf5j, memex-portal-deployment-85587f8869-prgnp, memex-portal-deployment-85587f8869-z9bqf, memex-portal-deployment-5499f76b76-sz46x, memex-portal-deployment-55fffcbc56-4zd9g, memex-portal-deployment-5499f76b76-ltlwc, memex-portal-deployment-55fffcbc56-knljr, memex-portal-deployment-74bfbdff9-xmfpc, memex-portal-deployment-6749fbfc59-mz4k5, memex-portal-deployment-6749fbfc59-p29vl
    Recent log lines 
    2026-10-11 03:58:39Z memex-portal-deployment-74bfbdff9-xmfpc fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.6.197:11111:150693988 sys.client/hosted-10.244.6.197:11111@150693988]->[ messagehub/Ops/Issue/control__job-memex-migration-roll-3-1-10372__JobFailed] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #1985E0E9D446C4C6
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Ops/Issue/control__job-memex-migration-roll-3-1-10372__JobFailed.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
        …[truncated]
2026-10-11 03:58:50Z memex-portal-deployment-74bfbdff9-xmfpc fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.6.197:11111:150693988 sys.client/hosted-10.244.6.197:11111@150693988]->[ messagehub/AppleWeather/Weather/Test/WeatherTestsArea] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #1985E0E9D446DD50
      System.TimeoutException: Grain placement operation timed out for grain messagehub/AppleWeather/Weather/Test/WeatherTestsArea.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.Ex…[truncated]
2026-10-11 03:59:19Z memex-portal-deployment-74bfbdff9-xmfpc fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.6.197:11111:150693988 sys.client/hosted-10.244.6.197:11111@150693988]->[ messagehub/AppleMaps/Maps/Source/AmapsConnect] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #1985E0E9D4462E78
      System.TimeoutException: Grain placement operation timed out for grain messagehub/AppleMaps/Maps/Source/AmapsConnect.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  18. systemorph-com commented on Oct 11, 2026

    @systemorph-com
    systemorph-combot
    on Oct 11, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 164 more since the last update (856 total).

    • Last seen: 2026-10-11 04:13:39Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-56fbdcd48f-7cd4h, memex-portal-deployment-56fbdcd48f-cbms9, memex-portal-deployment-555d9cd496-qntsh, memex-portal-deployment-54bd7f5466-vnxfz, memex-portal-deployment-57b7d56fdd-5wqxq, memex-portal-deployment-58bf75c9d8-zmq48, memex-portal-deployment-76dd69c7fc-xxql9, memex-portal-deployment-76dd69c7fc-6tjhj, memex-portal-deployment-76dd69c7fc-xzsdl, memex-portal-deployment-6cfcf78898-lsklh, memex-portal-deployment-65f6466887-gv9qs, memex-portal-deployment-6cdc9bf66f-mbfsl, memex-portal-deployment-5f6b9fc9b-hrdhq, memex-portal-deployment-845898d964-mwzl7, memex-portal-deployment-845898d964-ptszc, memex-portal-deployment-845898d964-f6wm5, memex-portal-deployment-7648c9c56f-7nf5j, memex-portal-deployment-85587f8869-prgnp, memex-portal-deployment-85587f8869-z9bqf, memex-portal-deployment-5499f76b76-sz46x, memex-portal-deployment-55fffcbc56-4zd9g, memex-portal-deployment-5499f76b76-ltlwc, memex-portal-deployment-55fffcbc56-knljr, memex-portal-deployment-74bfbdff9-xmfpc, memex-portal-deployment-6749fbfc59-mz4k5, memex-portal-deployment-6749fbfc59-p29vl
    Recent log lines 
    2026-10-11 04:13:39Z memex-portal-deployment-74bfbdff9-xmfpc fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.6.197:11111:150693988 sys.client/hosted-10.244.6.197:11111@150693988]->[ messagehub/AppleWeather/Weather/Test/WeatherTestsArea] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #1985E0E9D4446DBF
      System.TimeoutException: Grain placement operation timed out for grain messagehub/AppleWeather/Weather/Test/WeatherTestsArea.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.Ex…[truncated]
2026-10-11 04:13:39Z memex-portal-deployment-74bfbdff9-xmfpc fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.6.197:11111:150693988 sys.client/hosted-10.244.6.197:11111@150693988]->[ messagehub/AppleWeather/Weather/Test/WeatherTestsArea] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #1985E0E9D44410AC
      System.TimeoutException: Grain placement operation timed out for grain messagehub/AppleWeather/Weather/Test/WeatherTestsArea.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.Ex…[truncated]
2026-10-11 04:13:39Z memex-portal-deployment-74bfbdff9-xmfpc fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.6.197:11111:150693988 sys.client/hosted-10.244.6.197:11111@150693988]->[ messagehub/AppleWeather/Weather/Test/WeatherTestsArea] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #1985E0E9D44410A9
      System.TimeoutException: Grain placement operation timed out for grain messagehub/AppleWeather/Weather/Test/WeatherTestsArea.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.Ex…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  19. rbuergi commented on Oct 11, 2026

    @rbuergi
    rbuergi
    on Oct 11, 2026
    Contributor

    (re-posted 2026-10-11; originally by systemorph-com[bot] on 2026-09-24)

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 47 more since the last update (82 total).

    • Last seen: 2026-09-23 18:11:49Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv
    Recent log lines 
    2026-09-23 18:11:49Z memex-portal-deployment-7d7f4d84f9-wl8mv fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.240:11111:149191174 sys.client/hosted-10.244.3.240:11111@149191174]->[ messagehub/ClientFReporting/Valuation] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #4C9EADC2954D9202
      System.TimeoutException: Grain placement operation timed out for grain messagehub/ClientFReporting/Valuation.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallbac…[truncated]
2026-09-23 18:11:49Z memex-portal-deployment-7d7f4d84f9-wl8mv fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.240:11111:149191174 sys.client/hosted-10.244.3.240:11111@149191174]->[ messagehub/ClientFReporting/Valuation] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #4C9EADC2954D9124
      System.TimeoutException: Grain placement operation timed out for grain messagehub/ClientFReporting/Valuation.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallbac…[truncated]
2026-09-23 18:11:49Z memex-portal-deployment-7d7f4d84f9-wl8mv fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.240:11111:149191174 sys.client/hosted-10.244.3.240:11111@149191174]->[ messagehub/ClientFReporting/Fund] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #4C9EADC2954D9204
      System.TimeoutException: Grain placement operation timed out for grain messagehub/ClientFReporting/Fund.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  20. Systemorph
    deleted a comment from systemorph-com on Oct 11, 2026

  21. rbuergi commented on Oct 11, 2026

    @rbuergi
    rbuergi
    on Oct 11, 2026
    Contributor

    (re-posted 2026-10-11; originally by systemorph-com[bot] on 2026-09-26)

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 2 more since the last update (84 total).

    • Last seen: 2026-09-24 06:29:19Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8
    Recent log lines 
    2026-09-23 18:11:49Z memex-portal-deployment-7d7f4d84f9-wl8mv fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.240:11111:149191174 sys.client/hosted-10.244.3.240:11111@149191174]->[ messagehub/ClientFReporting/Fund] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #4C9EADC2954D9204
      System.TimeoutException: Grain placement operation timed out for grain messagehub/ClientFReporting/Fund.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback…[truncated]
2026-09-24 06:29:19Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.41:11111:149226787 sys.client/hosted-10.244.2.41:11111@149226787]->[ messagehub/Acp] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #3E90DCEC8F874F5C[ForwardCount=1]
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Acp.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state…[truncated]
2026-09-24 06:29:19Z memex-portal-deployment-7596dd76bb-cbbj8 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.41:11111:149226787 sys.client/hosted-10.244.2.41:11111@149226787]->[ messagehub/Acp] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #3E90DCEC8F874EA5[ForwardCount=1]
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Acp.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  22. Systemorph
    deleted a comment from systemorph-com on Oct 11, 2026

  23. rbuergi commented on Oct 11, 2026

    @rbuergi
    rbuergi
    on Oct 11, 2026
    Contributor

    (re-posted 2026-10-11; originally by systemorph-com[bot] on 2026-10-01)

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 9 more since the last update (256 total).

    • Last seen: 2026-10-01 12:08:45Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-56fbdcd48f-7cd4h, memex-portal-deployment-56fbdcd48f-cbms9, memex-portal-deployment-555d9cd496-qntsh, memex-portal-deployment-54bd7f5466-vnxfz, memex-portal-deployment-57b7d56fdd-5wqxq, memex-portal-deployment-58bf75c9d8-zmq48
    Recent log lines 
    2026-10-01 12:08:30Z memex-portal-deployment-57b7d56fdd-5wqxq fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.247:11111:149853811 sys.client/hosted-10.244.3.247:11111@149853811]->[ messagehub/rbuergi/MeshWeaverInstance/ci-crm] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #628C2E9C4B110C82
      System.TimeoutException: Grain placement operation timed out for grain messagehub/rbuergi/MeshWeaverInstance/ci-crm.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionC…[truncated]
2026-10-01 12:08:41Z memex-portal-deployment-57b7d56fdd-5wqxq fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.247:11111:149853811 sys.client/hosted-10.244.3.247:11111@149853811]->[ messagehub/Admin/SyncTokenSigningKey/current] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #628C2E9C4B110E19
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Admin/SyncTokenSigningKey/current.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionC…[truncated]
2026-10-01 12:08:45Z memex-portal-deployment-57b7d56fdd-5wqxq fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.247:11111:149853811 sys.client/hosted-10.244.3.247:11111@149853811]->[ messagehub/rbuergi/MeshWeaverInstance/client-b] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #628C2E9C4B11097C
      System.TimeoutException: Grain placement operation timed out for grain messagehub/rbuergi/MeshWeaverInstance/client-b.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionCon…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  24. Systemorph
    deleted a comment from systemorph-com on Oct 11, 2026

  25. rbuergi commented on Oct 11, 2026

    @rbuergi
    rbuergi
    on Oct 11, 2026
    Contributor

    (re-posted 2026-10-11; originally by systemorph-com[bot] on 2026-10-01)

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 3 more since the last update (259 total).

    • Last seen: 2026-10-01 12:08:54Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-56fbdcd48f-7cd4h, memex-portal-deployment-56fbdcd48f-cbms9, memex-portal-deployment-555d9cd496-qntsh, memex-portal-deployment-54bd7f5466-vnxfz, memex-portal-deployment-57b7d56fdd-5wqxq, memex-portal-deployment-58bf75c9d8-zmq48
    Recent log lines 
    2026-10-01 12:08:53Z memex-portal-deployment-58bf75c9d8-zmq48 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.229:11111:149861252 sys.client/hosted-10.244.2.229:11111@149861252]->[ messagehub/MeshWeaverInstance/03349f5b76b5] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #74522E43C1B7618B
      System.TimeoutException: Grain placement operation timed out for grain messagehub/MeshWeaverInstance/03349f5b76b5.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunConti…[truncated]
2026-10-01 12:08:53Z memex-portal-deployment-58bf75c9d8-zmq48 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.229:11111:149861252 sys.client/hosted-10.244.2.229:11111@149861252]->[ messagehub/MeshWeaverInstance/03349f5b76b5] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #74522E43C1B76275
      System.TimeoutException: Grain placement operation timed out for grain messagehub/MeshWeaverInstance/03349f5b76b5.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunConti…[truncated]
2026-10-01 12:08:54Z memex-portal-deployment-57b7d56fdd-5wqxq fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.247:11111:149853811 sys.client/hosted-10.244.3.247:11111@149853811]->[ messagehub/Admin/_PluginGrant/client-a] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #628C2E9C4B110BE2
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Admin/_PluginGrant/client-a.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, Co…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  26. Systemorph
    deleted a comment from systemorph-com on Oct 11, 2026

  27. rbuergi commented on Oct 11, 2026

    @rbuergi
    rbuergi
    on Oct 11, 2026
    Contributor

    (re-posted 2026-10-11; originally by systemorph-com[bot] on 2026-10-04)

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 2 more since the last update (261 total).

    • Last seen: 2026-10-04 19:10:55Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-56fbdcd48f-7cd4h, memex-portal-deployment-56fbdcd48f-cbms9, memex-portal-deployment-555d9cd496-qntsh, memex-portal-deployment-54bd7f5466-vnxfz, memex-portal-deployment-57b7d56fdd-5wqxq, memex-portal-deployment-58bf75c9d8-zmq48, memex-portal-deployment-76dd69c7fc-xxql9
    Recent log lines 
    2026-10-01 12:08:54Z memex-portal-deployment-57b7d56fdd-5wqxq fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.3.247:11111:149853811 sys.client/hosted-10.244.3.247:11111@149853811]->[ messagehub/Admin/_PluginGrant/client-a] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #628C2E9C4B110BE2
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Admin/_PluginGrant/client-a.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, Co…[truncated]
2026-10-04 19:10:55Z memex-portal-deployment-76dd69c7fc-xxql9 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.132:11111:150139264 sys.client/hosted-10.244.2.132:11111@150139264]->[ messagehub/Admin/_LogIncident/9fa365a8221fa425] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #74FCC7971EDA9216
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Admin/_LogIncident/9fa365a8221fa425.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext execut…[truncated]
2026-10-04 19:10:55Z memex-portal-deployment-76dd69c7fc-xxql9 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.2.132:11111:150139264 sys.client/hosted-10.244.2.132:11111@150139264]->[ messagehub/Admin/_LogIncident/9fa365a8221fa425] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #74FCC7971EDA93E9
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Admin/_LogIncident/9fa365a8221fa425.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext execut…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  28. Systemorph
    deleted a comment from systemorph-com on Oct 11, 2026

  29. systemorph-com commented on Oct 11, 2026

    @systemorph-com
    systemorph-combot
    on Oct 11, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 2 more since the last update (858 total).

    • Last seen: 2026-10-11 22:10:50Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-56fbdcd48f-7cd4h, memex-portal-deployment-56fbdcd48f-cbms9, memex-portal-deployment-555d9cd496-qntsh, memex-portal-deployment-54bd7f5466-vnxfz, memex-portal-deployment-57b7d56fdd-5wqxq, memex-portal-deployment-58bf75c9d8-zmq48, memex-portal-deployment-76dd69c7fc-xxql9, memex-portal-deployment-76dd69c7fc-6tjhj, memex-portal-deployment-76dd69c7fc-xzsdl, memex-portal-deployment-6cfcf78898-lsklh, memex-portal-deployment-65f6466887-gv9qs, memex-portal-deployment-6cdc9bf66f-mbfsl, memex-portal-deployment-5f6b9fc9b-hrdhq, memex-portal-deployment-845898d964-mwzl7, memex-portal-deployment-845898d964-ptszc, memex-portal-deployment-845898d964-f6wm5, memex-portal-deployment-7648c9c56f-7nf5j, memex-portal-deployment-85587f8869-prgnp, memex-portal-deployment-85587f8869-z9bqf, memex-portal-deployment-5499f76b76-sz46x, memex-portal-deployment-55fffcbc56-4zd9g, memex-portal-deployment-5499f76b76-ltlwc, memex-portal-deployment-55fffcbc56-knljr, memex-portal-deployment-74bfbdff9-xmfpc, memex-portal-deployment-6749fbfc59-mz4k5, memex-portal-deployment-6749fbfc59-p29vl, memex-portal-deployment-b7bdc4988-f275z
    Recent log lines 
    2026-10-11 04:13:39Z memex-portal-deployment-74bfbdff9-xmfpc fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.6.197:11111:150693988 sys.client/hosted-10.244.6.197:11111@150693988]->[ messagehub/AppleWeather/Weather/Test/WeatherTestsArea] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #1985E0E9D44410A9
      System.TimeoutException: Grain placement operation timed out for grain messagehub/AppleWeather/Weather/Test/WeatherTestsArea.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.Ex…[truncated]
2026-10-11 22:10:50Z memex-portal-deployment-b7bdc4988-f275z fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.6.121:11111:150759605 sys.client/hosted-10.244.6.121:11111@150759605]->[ messagehub/SwissSolvencyTest/_Activity/a65f7f11] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #11564521CFB7DBCE
      System.TimeoutException: Grain placement operation timed out for grain messagehub/SwissSolvencyTest/_Activity/a65f7f11.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TS…[truncated]
2026-10-11 22:10:50Z memex-portal-deployment-b7bdc4988-f275z fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.6.121:11111:150759605 sys.client/hosted-10.244.6.121:11111@150759605]->[ messagehub/SwissSolvencyTest/_Activity/a65f7f11] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #11564521CFB7E2E3
      System.TimeoutException: Grain placement operation timed out for grain messagehub/SwissSolvencyTest/_Activity/a65f7f11.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TS…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

  30. systemorph-com commented on Oct 12, 2026

    @systemorph-com
    systemorph-combot
    on Oct 12, 2026
    ContributorAuthor

    Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.

    Still occurring: 6 more since the last update (864 total).

    • Last seen: 2026-10-11 22:12:55Z
    • Pods: memex-portal-deployment-89868db67-xg22d, memex-portal-deployment-5c444645f8-8pdzb, memex-portal-deployment-5f95fb94dc-jvgv2, memex-portal-deployment-5576ddf6f4-c5mz2, memex-portal-deployment-777677694-p6hbn, memex-portal-deployment-7d7f4d84f9-mvzb5, memex-portal-deployment-7d7f4d84f9-n7g6b, memex-portal-deployment-7d7f4d84f9-wl8mv, memex-portal-deployment-7596dd76bb-cbbj8, memex-portal-deployment-76cf776847-v4txp, memex-portal-deployment-56fbdcd48f-4dpcn, memex-portal-deployment-56fbdcd48f-sz5st, memex-portal-deployment-745745db7c-b2gjn, memex-portal-deployment-56fbdcd48f-7cd4h, memex-portal-deployment-56fbdcd48f-cbms9, memex-portal-deployment-555d9cd496-qntsh, memex-portal-deployment-54bd7f5466-vnxfz, memex-portal-deployment-57b7d56fdd-5wqxq, memex-portal-deployment-58bf75c9d8-zmq48, memex-portal-deployment-76dd69c7fc-xxql9, memex-portal-deployment-76dd69c7fc-6tjhj, memex-portal-deployment-76dd69c7fc-xzsdl, memex-portal-deployment-6cfcf78898-lsklh, memex-portal-deployment-65f6466887-gv9qs, memex-portal-deployment-6cdc9bf66f-mbfsl, memex-portal-deployment-5f6b9fc9b-hrdhq, memex-portal-deployment-845898d964-mwzl7, memex-portal-deployment-845898d964-ptszc, memex-portal-deployment-845898d964-f6wm5, memex-portal-deployment-7648c9c56f-7nf5j, memex-portal-deployment-85587f8869-prgnp, memex-portal-deployment-85587f8869-z9bqf, memex-portal-deployment-5499f76b76-sz46x, memex-portal-deployment-55fffcbc56-4zd9g, memex-portal-deployment-5499f76b76-ltlwc, memex-portal-deployment-55fffcbc56-knljr, memex-portal-deployment-74bfbdff9-xmfpc, memex-portal-deployment-6749fbfc59-mz4k5, memex-portal-deployment-6749fbfc59-p29vl, memex-portal-deployment-b7bdc4988-f275z, memex-portal-deployment-5b769df8cb-xmkk5
    Recent log lines 
    2026-10-11 22:12:24Z memex-portal-deployment-5b769df8cb-xmkk5 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.214:11111:150760299 sys.client/hosted-10.244.5.214:11111@150760299]->[ messagehub/RiskTransfer/_Activity/6943a761] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #7617021ABAAB0EE9
      System.TimeoutException: Grain placement operation timed out for grain messagehub/RiskTransfer/_Activity/6943a761.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func…[truncated]
2026-10-11 22:12:55Z memex-portal-deployment-5b769df8cb-xmkk5 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.214:11111:150760299 sys.client/hosted-10.244.5.214:11111@150760299]->[ messagehub/Governance/_Activity/c01a7ae2] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #7617021ABAAB1E1B
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Governance/_Activity/c01a7ae2.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 c…[truncated]
2026-10-11 22:12:55Z memex-portal-deployment-5b769df8cb-xmkk5 fail: Orleans.Messaging[100071]
      Failed to address message Request [S10.244.5.214:11111:150760299 sys.client/hosted-10.244.5.214:11111@150760299]->[ messagehub/Governance/_Activity/c01a7ae2] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #7617021ABAAB1C31
      System.TimeoutException: Grain placement operation timed out for grain messagehub/Governance/_Activity/c01a7ae2.
       ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
       ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 602
         at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
         --- End of inner exception stack trace ---
         at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
         at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
         at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
         at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
         at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
         at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
         at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 c…[truncated]

    Re-addressed by the current identity function: this incident inherited c4e121bf40d7f9f6 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingsev:LLow - cosmetic, rare edge case, or developer-only

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions