Repository navigation
Pod-hub startup claim fires before any silo is Active, failing Attach placement with an unclassified OrleansException #3983
Description
What is failing
The pod-hub claim (OrleansRoutingService.RegisterStream → IPodHubGrain.Attach) is issued at startup without being ordered on silo readiness. When it fires before the local silo (or any silo in the cluster) has declared the podhub grain type, Orleans placement cannot find a compatible silo and throws OrleansException: No active nodes are compatible with grain podhub … Known nodes with grain type: none — the Attach call dies at addressing time and the hub is unreachable until the next membership change re-asserts the claim.
Probable cause — high confidence.
This is the startup-population half of the fault documented in The Pod-Hub Claim Must Be Re-Asserted → "What this does not fix": RegisterStream orders its Orleans stream subscription on OrleansStreamingReadiness (lifecycle stage Active) but issues the pod-hub claim immediately, unordered. The samples show two pods of the same deployment (memex-portal-deployment-5867566569-*) each bursting within a single minute at 21:01–21:02, against both mesh/{id} and cache/{id} hubs — exactly the eager-registration burst the doc predicts, now failing with a placement exception rather than PodHubNotHereException (which only occurs once an activation exists to answer). Per the doc, the membership change on reaching Active repairs the mapping, so the outage window is bounded by the retry/re-assertion — but the wasted burst, and its error-level log, remain. Two secondary observations: (a) this exception shape is neither PodHubNotHereException nor covered by IsDirectoryUnstable/IsTransientFailure classification, so it is an unclassified failure in the claim path; (b) the doc already names closing this as "a separate, smaller change" — this ticket is that change.
Impact
34 occurrences in ~65 s across two pods, both within the deploy/startup window. Bounded, self-repairing, no evidence of the permanent lost-mapping state (that fault is closed). Real cost is a hard-error burst per pod startup and a startup window in which pod hubs are unaddressable.
Where to look
OrleansRoutingService.RegisterStream / AttachPodHub (MeshWeaver.Connection.Orleans): gate the pod-hub claim on the same readiness stage the stream subscription already uses (OrleansStreamingReadiness / lifecycle Active), and/or classify this placement OrleansException as transient in the claim's failure handling. Context: PodHubDeliveryRollPlan, PodHubClaimReassertion. Likely related: #2901, #2357.
Evidence
| Fingerprint | 160571be8544318a |
| Category | Orleans.Messaging |
| Severity | Error |
| Exception | Orleans.Runtime.OrleansException |
| Top frame | Polly.Outcome1.ThrowIfException()` |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-5867566569-k9nmf, memex-portal-deployment-5867566569-frfqx |
| Occurrences | 34 |
| First seen | 2026-09-10 21:01:27Z |
| Last seen | 2026-09-10 21:02:32Z |
Recent log lines
2026-09-10 21:01:28Z memex-portal-deployment-5867566569-k9nmf fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.3.184:11111:148078886 sys.client/hosted-10.244.3.184:11111@148078886]->[ podhub/mesh/U3qb_hjKbkeWh5AshBZ4EQ] MeshWeaver.Connection.Orleans.IPodHubGrain.Attach() #69DC6F6D9B4ECBDE
Orleans.Runtime.OrleansException: No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 221
at Orleans.Runtime.Placement.PreferLocalPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/PreferLocalPlacementDirector.cs:line 21
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 612
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
at Polly.Outcome`1.ThrowIfException()
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 540
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 491
--- End of stack trace from previous location ---
at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/Me…[truncated]
2026-09-10 21:01:28Z memex-portal-deployment-5867566569-k9nmf fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.3.184:11111:148078886 sys.client/hosted-10.244.3.184:11111@148078886]->[ podhub/mesh/U3qb_hjKbkeWh5AshBZ4EQ] MeshWeaver.Connection.Orleans.IPodHubGrain.Attach() #69DC6F6D9B4ECBC9
Orleans.Runtime.OrleansException: No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 221
at Orleans.Runtime.Placement.PreferLocalPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/PreferLocalPlacementDirector.cs:line 21
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 612
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
at Polly.Outcome`1.ThrowIfException()
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 540
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 491
--- End of stack trace from previous location ---
at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/Me…[truncated]
2026-09-10 21:01:28Z memex-portal-deployment-5867566569-k9nmf fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.3.184:11111:148078886 sys.client/hosted-10.244.3.184:11111@148078886]->[ podhub/mesh/U3qb_hjKbkeWh5AshBZ4EQ] MeshWeaver.Connection.Orleans.IPodHubGrain.Attach() #69DC6F6D9B4ECBCB
Orleans.Runtime.OrleansException: No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 221
at Orleans.Runtime.Placement.PreferLocalPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/PreferLocalPlacementDirector.cs:line 21
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 612
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
at Polly.Outcome`1.ThrowIfException()
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 540
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 491
--- End of stack trace from previous location ---
at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/Me…[truncated]
2026-09-10 21:02:32Z memex-portal-deployment-5867566569-frfqx fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.2.20:11111:148078951 sys.client/hosted-10.244.2.20:11111@148078951]->[ podhub/cache/oKwC1pk5zkqnYpfsDBYHwQ] MeshWeaver.Connection.Orleans.IPodHubGrain.Attach() #3F042DAF4F3A6654
Orleans.Runtime.OrleansException: No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 221
at Orleans.Runtime.Placement.PreferLocalPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/PreferLocalPlacementDirector.cs:line 21
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 612
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
at Polly.Outcome`1.ThrowIfException()
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 540
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 491
--- End of stack trace from previous location ---
at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/Mes…[truncated]
2026-09-10 21:02:32Z memex-portal-deployment-5867566569-frfqx fail: Orleans.Messaging[100071]
Failed to address message Request [S10.244.2.20:11111:148078951 sys.client/hosted-10.244.2.20:11111@148078951]->[ podhub/mesh/oKwC1pk5zkqnYpfsDBYHwQ] MeshWeaver.Connection.Orleans.IPodHubGrain.Attach() #3F042DAF4F3A6657
Orleans.Runtime.OrleansException: No active nodes are compatible with grain podhub and interface MeshWeaver.Connection.Orleans.IPodHubGrain version 0. Known nodes with grain type: none. All known nodes compatible with interface version: none
at Orleans.Runtime.Placement.PlacementService.GetCompatibleSilos(PlacementTarget target) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 221
at Orleans.Runtime.Placement.PreferLocalPlacementDirector.OnAddActivation(PlacementStrategy strategy, PlacementTarget target, IPlacementContext context) in /_/src/Orleans.Runtime/Placement/PreferLocalPlacementDirector.cs:line 21
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.ExecutePlacementAsync(PlacementContext context, CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 612
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.<ExecutePlacementOutcomeAsync>g__AwaitOutcome|21_0(ValueTask`1 pending) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 576
at Polly.Outcome`1.ThrowIfException()
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 540
at Orleans.Runtime.Placement.PlacementService.PlacementWorker.AddressWaitingMessages(GrainPlacementWorkItem completedWorkItem) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 491
--- End of stack trace from previous location ---
at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync|42_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/Mess…[truncated]
Opened automatically from Admin/_LogIncident/160571be8544318a. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 11, 2026
Implemented in PR #3988 and shipped to production as 3.0.0-ci.8323. Production verification on memex-cloud: the Kubernetes rollout completed successfully; a live audit recorded the running image as meshweaver.azurecr.io/memex-portal-ai:3.0.0-ci.8323; the public health endpoint returned HTTP 200; and a 60-minute post-rollout log query for fingerprint 160571be8544318a plus the exact pod-hub placement failure returned 0 lines. Closing as completed.
fix(#3931): the pod-hub claim dropped membership changes in two windows
systemorph-com commentedon Sep 17, 2026
Reopened — this fault is happening again.
Still occurring: 147 more since the last update (181 total).
memex-portal-deployment-5867566569-k9nmf,memex-portal-deployment-5867566569-frfqx,memex-portal-deployment-5867566569-c9xbt,memex-portal-deployment-5867566569-z4zzk,memex-portal-deployment-7f74766b8d-rltnt,memex-portal-deployment-7f74766b8d-rmjfc,memex-portal-deployment-67f7fc74f6-hxmct,memex-portal-deployment-67f7fc74f6-cqwr5,memex-portal-deployment-67f7fc74f6-cvvnq,memex-portal-deployment-574d54c98d-cdd47,memex-portal-deployment-574d54c98d-s8g9b,memex-portal-deployment-574d54c98d-vvcwqRecent log lines
rbuergi commentedon Sep 17, 2026
Re-closed — the reopen above was on stale evidence (#4577).
The recurrence bot reopened this at
2026-09-17T07:52:24Zunder the headline "this fault is happening again", but the occurrence its own comment cites is OLDER than the close it reverted:Last seen:2026-09-11 03:48:35Z2026-09-11T05:24:11ZSo the counter moved, not the clock — the fault's newest occurrence predates the close, which is the one thing "still happening" has to mean. Root cause and fix:
LogIncidentFiler.Reopenreopened on the sole condition that the issue was closed, while the comment-debt clock pays a final count for incidents whose fault has stopped (the fix, Systemorph/MeshWeaver.Plugins#2014, is written but NOT yet merged — it makes a reopen requireincident.LastSeen > issue.ClosedAt).This restores the close exactly as it stood; nothing is asserted about the defect itself. If the fault happens again, the next occurrence postdates this close and reopens the issue on evidence that means what it says.
One of 53 issues whose reopen was measured stale (129 bot reopens on 2026-09-17, 125 of them inside the 07:51-07:53Z sweep; 76 postdate their close and were left open). Measured and re-closed 2026-09-17 by Claude Opus 5.