Repository navigation
Grain placement for a message-hub grain times out, dropping DeliverMessage — recurrence of #4672 for messagehub/mkleiner/_Install/LinkedIn #5340
Description
What is failing
Message delivery from the portal's hosted client to a single message-hub grain (messagehub/mkleiner/_Install/LinkedIn) fails because Orleans cannot complete grain placement for that grain within the 30-second Polly timeout, so the IMessageHubGrain.DeliverMessage request never reaches the grain. This is the same defect already filed as #4672 from incident Admin/_LogIncident/67ee310417b64739 — same log site, same exception chain (TimeoutException → TimeoutRejectedException → TaskCanceledException in the Orleans placement pipeline).
Probable cause
Low confidence, and unchanged from the earlier analysis: the failure is in Orleans' own placement pipeline (PlacementService.PlacementWorker.ExecutePlacementAsync), not MeshWeaver code. Placement timing out for one specific grain on an otherwise healthy pod points at the target silo being unable to complete placement/directory lookup for that grain at that moment — e.g. the hub for the mkleiner/_Install/LinkedIn space being activated or blocked — or transient cluster-membership churn around 2026-09-22T14:41Z. Not determinable from the client-side log alone; correlate with silo-side Orleans logs for that window.
Impact
Minimal on its own: 2 occurrences in a single second on one pod, all targeting the same grain — a brief window in which messages to that one hub were dropped/timed out. However, this is the fourth day this fingerprint has recurred (Sep 17–20 cluster of 139 occurrences across 9 pods in #4672, now reappearing Sep 22), so it is not transient noise — the underlying placement bottleneck is still live.
Where to look
MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage— the failing request path.- Orleans
PlacementService.cs(ExecutePlacementAsync) — the placement worker that timed out. - Grain
messagehub/mkleiner/_Install/LinkedInaround 2026-09-22T14:41:52Z; check whether the hub was being activated/recycled at that moment.
Duplicate note: this instance should likely be closed as a duplicate of #4672 rather than triaged separately — filed here so the recurrence (a previously unseen grain, after two quiet days) is on the record for whoever owns that issue.
Evidence
| Fingerprint | 4f4433e236df5eaa |
| Category | Orleans.Messaging |
| Severity | Error |
| Exception | System.TimeoutException |
| Top frame | Polly.Retry.RetryResilienceStrategy1.ExecuteCore[TState](Func3 callback, ResilienceContext context, TState state) |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-5c444645f8-8pdzb |
| Occurrences | 2 |
| First seen | 2026-09-22 14:41:52Z |
| Last seen | 2026-09-22 14:41:52Z |
| Routing | not determined — no configured route matches the category Orleans.Messaging. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-22 14:41:52Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/mkleiner/_Install/LinkedIn] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B484517
System.TimeoutException: Grain placement operation timed out for grain messagehub/mkleiner/_Install/LinkedIn.
---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
--- End of inner exception stack trace ---
at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.Execut…[truncated]
2026-09-22 14:41:52Z memex-portal-deployment-5c444645f8-8pdzb fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.5.129:11111:149091044 sys.client/hosted-10.244.5.129:11111@149091044]->[ messagehub/mkleiner/_Install/LinkedIn] MeshWeaver.Connection.Orleans.IMessageHubGrain.DeliverMessage(MeshWeaver.Messaging.IMessageDelivery) #662B61156B484C42
System.TimeoutException: Grain placement operation timed out for grain messagehub/mkleiner/_Install/LinkedIn.
---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
--- End of inner exception stack trace ---
at Polly.Utils.ExceptionUtilities.TrySetStackTrace[T](T exception)
at Polly.Timeout.TimeoutResilienceStrategy.ExecuteCore[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at Polly.Retry.RetryResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.Execut…[truncated]
Opened automatically from Admin/_LogIncident/4f4433e236df5eaa. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 23, 2026
Duplicate of #5336, the one silo departure on memex-cloud at 2026-09-22 14:41Z.
Grain placement operation timed outformessagehub/mkleiner/_Install/LinkedInon5c444645f8-8pdzbat 14:41:52Z. That pod was logging refusals toS10.244.9.90:11111:149090968from 14:41:30 to 14:42:06Z, and a lookup issued at about 14:41:22 into the departing silo's directory partition expires exactly then. So this sample belongs to the departure, not to #4672/#5037's cluster-wide starvation reading. #4672's own history is left as the maintainer grouped it.The shared site and discriminator: an Orleans directory lookup or placement into the partition of the departed silo
S10.244.9.90:11111:149090968, during its ~36 s window, on the same two survivor pods, with no MeshWeaver frame. The grouping, what MeshWeaver does with the rejectedDeliverMessage, and the one read still owed (graceful stop vs. crash) are on #5336 (#5336 (comment)) and in PR #5450.systemorph-com commentedon Sep 23, 2026
Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.
Still occurring: 2 more since the last update (4 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-7d7f4d84f9-n7g6bRecent log lines
systemorph-com commentedon Sep 26, 2026
Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.
Still occurring: 4 more since the last update (8 total).
memex-portal-deployment-5c444645f8-8pdzb,memex-portal-deployment-7d7f4d84f9-n7g6b,memex-portal-deployment-745745db7c-b2gjnRecent log lines