Grain calls are dropped instead of retried when the Orleans directory is unstable during a rollout #2357

Activity

1. rbuergi commented on Aug 26, 2026

   

   rbuergi

   on Aug 26, 2026

   Contributor

   More actions

   Narrowed against origin/main. #2314 (merged today, 07:22Z) fixes an adjacent class, not this one — and half of this issue's evidence is outside any MeshWeaver retry path entirely.

   The four call sites split into two groups

   Group A — MeshWeaver's own grains (IMessageHubGrain.DeliverMessage, IPodHubGrain.Deliver), routed through RoutingGrain. RoutingGrain.IsTransientFailure (src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs:1076) is the sole retry gate for both:

   internal static bool IsTransientFailure(Exception ex) =>
       ex is TimeoutException
           or global::Orleans.Runtime.OrleansMessageRejectionException
       || (ex.InnerException != null && IsTransientFailure(ex.InnerException));

   This issue's stack traces name the exception type explicitly: Orleans.Runtime.OrleansException — the base type, not OrleansMessageRejectionException — thrown directly out of LocalGrainDirectory.LookupAsync ("cannot forward LookUpAsync … because hop limit is reached", "not stable to perform the lookup … Retry later"). Pattern is OrleansMessageRejectionException does not match a plain OrleansException instance, so IsTransientFailure returns false for exactly this failure shape — it propagates on the first attempt regardless of #2314.

   #2314 does not touch this. Its own PR body confirms the fix is scoped to BuildPodHubRoute gaining the same IsTransientFailure-gated retry BuildGrainRoute already had — same classifier, same gap.

   Group B — Orleans-internal streaming grains (IPubSubRendezvousGrain.RegisterProducer, IMemoryStreamQueueGrain.Enqueue/Dequeue). grep -rn "IPubSubRendezvousGrain\|IMemoryStreamQueueGrain" src/ returns zero direct callers — these are invoked by Orleans' own Memory stream provider machinery (sys.svc.stream.agent, sys.client in the log's sender identity), entirely outside RoutingGrain's retry path. No MeshWeaver-level fix can touch these without changing the stream provider itself.

   This is very likely the same root as #1742's still-open residual A

   Group B's exception ("Current directory … not stable … it maps to S…, which is not a valid silo") is a MemoryStreamQueueGrain losing its directory entry during a silo departure mid-rollout — exactly the shape #1742's latest disposition (2026-08-25) scoped as residual A: "a message in flight can die with the silo hosting its queue grain during a roll." That comment already establishes the fix needs a second, durable-but-scoped stream provider and flags Microsoft.Orleans.Streaming.AdoNet as pre-release-only — a decision not yet made.

   Recommendation

   Not a duplicate to close as such (different symptom text, and Group A's directory-lookup gap is a smaller, distinct fix from Group B), but the durable resolution for Group B is the same maintainer decision #1742 is already blocked on. Group A (IsTransientFailure widening to cover plain OrleansException from a directory lookup, if that is judged safe/transient) is a small, separable fix that does not require that decision — worth splitting out if picked up.

   Leaving open; impact remains low-volume/self-clearing per the issue's own note.
2. rbuergi commented on Aug 27, 2026

   

   rbuergi

   on Aug 27, 2026

   Contributor

   More actions

   Defect item 1 is confirmed and fixed in #2518 — the calls really were dropped, and here is why nothing retried them

   Your "medium confidence" question was exactly right, and the answer is no, they were not retried. The mechanism, from Orleans' own source:

   MessageCenter.OnAddressingFailure
     → RejectMessage(msg, RejectionTypes.Unrecoverable, ex)      ← the DIRECTORY's exception, ATTACHED
     → caller: CallbackData.HandleRejectionResponse
           exception = rejection?.Exception ?? new OrleansMessageRejectionException(…)
                       ^^^^^^^^^^^^^^^^^^^^ the carried one WINS
   

   Because the rejection carries its own exception, the caller never sees the wrapper. RoutingGrain.IsTransientFailure matched TimeoutException and OrleansMessageRejectionException; what actually arrives is a bare Orleans.Runtime.OrleansException — the one your log shows, whose sibling variant literally ends "Retry later." (verified as a string literal in the pinned Orleans 10.2.2 Orleans.Runtime.dll).

   So DeliverToGrainObservable rethrew on the first attempt, and then both exception arms NACK'd the sender with a hard-coded terminal ErrorType.Failed. That second half is the more expensive one: SynchronizationStream's resubscribe latch and MeshNodeStreamCache.IsTransientOwnerFailure ride out ShuttingDown and tear down on Failed — so a live mirror that would have resumed seconds after the handover was killed permanently. And when PostFailure's own directed NACK hit the same unstable directory, it had no fallback at all and the sender was told nothing.

   #2518 fixes it as a classification change, not a new retry: OrleansRoutingService.IsDirectoryUnstable makes the existing bounded retry-with-fresh-resolve reachable, RoutingGrain.ClassifyDeliveryException replaces both terminal arms, and PostFailure falls back to the stream on any failed directed NACK. The marker is pinned against the shipped Orleans assembly's string literals, so an Orleans upgrade that rewords the message fails RED rather than silently reverting the classifier to inert.

   Your item 2 is NOT covered, and it is a bigger call than it looks

   "Why the in-memory stream provider is in use in a multi-silo cloud deployment at all" is the standing decision on #1742 — and worth knowing before anyone acts on it: Microsoft.Orleans.Streaming.AdoNet has never shipped a stable version, so it is a decision to ship a pre-release package in the reply path, not a config change.

   Also worth recording against your framing: since the pod-hub transport swap, the memory stream is only the fallback for delivery to a pod-process hub (the roll window, and hubs owned by an Orleans client process, which cannot host a grain). The pubsubrendezvous/… and memorystreamqueue/… entries in your own evidence are that fallback plus Orleans' internal streaming, not the primary path. That shrinks item 2's blast radius considerably — see #1742 for the full reasoning.

   Leaving this open on item 2.
3. rbuergi commented on Aug 27, 2026

   

   rbuergi

   on Aug 27, 2026

   Contributor

   More actions

   The durable-stream-provider decision has been re-sized — please read before acting on it

   #1742 was investigated today and the finding changes the scope of the pre-release Microsoft.Orleans.Streaming.AdoNet decision, which had been framed as "five tickets, one root".

   It is not. The primary reply path never goes near the stream provider — measured with the router's own trace (MESHWEAVER_MSG_TRACE=1): a two-silo PingRequest issued on mesh/{id} returned POD_HUB_OK for its reply leg, delivered by directed grain call and processed at the root hub. The durable provider improves only the fallback (queue-grain RAM, Orleans-client-hosted hubs).

   issue	covered by the durable provider?
   #2357	NO — covered by the CLASSIFIER fix instead. Its defect item 1 verbatim. Item 2 (memory streams at all) is the provider decision.
   #2321	PARTLY — requester now retried and correctly classified; the LogLineParser truncation half is untouched. Also looks like a collision bucket sharing #2357's events.
   #2320	YES, and only this — stream leg. Blast radius shrank to fallback-only.
   #2322	YES, and only this — stream leg. Same shrink.

   So size the provider decision on #2320 and #2322 alone.

   What actually caused the silent drops

   A classifier gap, not a transport one. The transport swap moved the reply leg onto Orleans' own grain directory — exactly what is unstable during a rolling deploy, which is the window every symptom was measured in.

   From Orleans' source: MessageCenter.OnAddressingFailure → RejectMessage(…, Unrecoverable, ex) with the directory's exception attached, and CallbackData.HandleRejectionResponse resolves rejection?.Exception ?? new OrleansMessageRejectionException(…) — the carried one wins. The router therefore receives a bare Orleans.Runtime.OrleansException whose own text ends "Retry later." (confirmed as a literal in the pinned Orleans 10.2.2 dll), while RoutingGrain.IsTransientFailure matched only TimeoutException and OrleansMessageRejectionException.

   Three consequences chained from that one gap:

   1. never retried;
   2. both exception arms NACK'd terminal ErrorType.Failed, which makes SynchronizationStream's latch tear down;
   3. PostFailure's own directed NACK hit the same unstable directory and reached LogUndeliverableNack — total silence.

   Prod evidence: 16 occurrences across two rolls.

   Two stale premises corrected, in case they are load-bearing elsewhere

   • mesh IS a stream-routed address type (MeshConfiguration.cs:68, since fix: /api/content hung ~50% of the time on multi-replica portals (#1729) #1736). The "mesh is not stream-routed — the OrleansCrossSiloReplyTest black hole" framing is out of date.
   • The MessageHubGrain.DeliverMessage TCS candidate (a TaskCompletionSource whose completion appeared to depend on OnDeactivateAsync) is bounded — BuildActivationChain's Amb timer terminates _hubReadyRaw at 30 s on every path except an enrichment that hangs unbounded, which is a separate compile-side concern.

   Not yet settled, and it gates closing #1742

   mesh/{id} is the only stream-routed hub claimed from an IHostedService at host start, and AttachPodHub has a ~3 s claim budget beside a stream subscription gated on 120 s of streaming readiness. A claim that fails to land degrades silently back to the stream. It does not reproduce in-process; #2518 adds the assertion that would catch it, and checking a live pod's AttachPodHub outcome is the honest next step.
4. rbuergi commented on Aug 29, 2026

   

   rbuergi

   on Aug 29, 2026

   Contributor

   More actions

   Fixed by #2518 (76a32acb0) — verified in the code on origin/main, not from the PR title

   Defect item 1a — the calls are retried. src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs:1376 IsTransientFailure now reaches the bare OrleansException, and names this issue in place:

   "🚨 THE BARE OrleansException THE TYPE TEST ABOVE CANNOT SEE — issue #1742 / #2357. Orleans rejects an un-addressable call with the DIRECTORY's exception attached… so a grain call made while the directory is mid-handoff arrives here as…"

   feeding the pre-existing retry-with-fresh-resolve loop.

   Defect item 1b — the NACK is no longer hard-coded terminal. RoutingGrain.cs:1422:

   internal static ErrorType ClassifyDeliveryException(Exception ex, Func<bool>? scopeDisposed = null) =>
       OrleansRoutingService.IsDirectoryUnstable(ex) || IsShutdownShaped(ex) || IsScopeTeardown(ex, scopeDisposed)
           ? ErrorType.ShuttingDown : ErrorType.Failed;

   so SynchronizationStream's resubscribe latch rides the handover out instead of tearing down.

   Both of your reported phrases are pinned, src/MeshWeaver.Connection.Orleans/OrleansRoutingService.cs:588-610:

   • DirectoryRetryLaterMarker = "Retry later"
   • DirectoryHopLimitMarker = "hop limit is reached" — documented as "the shape prod logged during two rolling deploys (issue Grain calls are dropped instead of retried when the Orleans directory is unstable during a rollout #2357)"

   with AggregateException/InnerException recursion at :596-597.

   Pinned by tests (test/MeshWeaver.Hosting.Orleans.Test/OrleansDirectoryInstabilityClassificationTest.cs): :108 asserts the call is made 3 times and does not end Failed; :139 an exhausted budget still NACKs transient; :92 an ordinary OrleansException stays terminal (non-vacuity); :203 RetryLaterMarker_IsStillTheShippedOrleansWording re-reads the phrase out of the pinned Orleans assembly, so an upgrade turns the classifier red rather than silently inert.

   Defect item 2 is not orphaned — the "memory streams at all" question was deliberately re-scoped onto #2320 and #2322, both still open, per the 2026-08-27 sizing comment.

   Not claimed: no post-fix production recurrence count was measured (the pre-fix evidence was 16 occurrences across two rolls). If the shape recurs, that is new evidence against a classifier that is otherwise demonstrably correct and guarded.

   Closing.
5. rbuergi

   closed this as completedon Aug 29, 2026
6. systemorph-com

   reopened this on Sep 17, 2026
7. systemorph-com commented on Sep 17, 2026

   

   systemorph-combot

   on Sep 17, 2026

   Contributor

   More actions

   Reopened — this fault is happening again.

   Still occurring: 26 more since the last update (42 total).

   • Last seen: 2026-09-10 12:26:14Z
   • Pods: memex-portal-deployment-7d6b864b54-w4xnl, memex-portal-deployment-558d86bb48-vl44h, memex-portal-deployment-9bf79b774-bwgqp, memex-portal-deployment-67dbc7bc9c-lswwx, memex-portal-deployment-9bf79b774-nvnzz, memex-portal-deployment-77cfc55bfc-q4f9r, memex-portal-deployment-77cfc55bfc-767mq, memex-portal-deployment-74bfcd8f6b-gkz48, memex-portal-deployment-74bfcd8f6b-sksm4, memex-portal-deployment-7456f8f447-lgtrr

   Recent log lines

   2026-09-07 11:58:22Z memex-portal-deployment-74bfcd8f6b-sksm4 fail: Orleans.Messaging[100071]
         Failed to address message Request [S10.244.2.22:11111:147785622 sys.client/hosted-10.244.2.22:11111@147785622]->[ podhub/portal/nodeops-fMEHZgBeDE-fQxvBO5Q4oA] MeshWeaver.Connection.Orleans.IPodHubGrain.Attach() #7E91FF6F74043A97
         Orleans.Runtime.OrleansException: Current directory at S10.244.5.41:11111:147772410 is not stable to perform the lookup for grainId podhub/portal/nodeops-fMEHZgBeDE-fQxvBO5Q4oA (it maps to S10.244.5.124:11111:147785772, which is not a valid silo). Retry later.
            at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 890
            at Orleans.Runtime.TaskRequest`1.Invoke() in /_/src/Orleans.Core.Abstractions/Runtime/GrainReference.cs:line 714
         --- End of stack trace from previous location ---
            at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Core/Core/GrainMethodInvoker.cs:line 133
            at MeshWeaver.Hosting.Orleans.AccessContextGrainCallFilter.Invoke(IIncomingGrainCallContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.Orleans/AccessContextGrainCallFilter.cs:line 38
            at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Core/Core/GrainMethodInvoker.cs:line 95
            at Orleans.Runtime.InsideRuntimeClient.Invoke(IGrainContext target, Message message) in /_/src/Orleans.Runtime/Core/InsideRuntimeClient.cs:line 318
         --- End of stack trace from previous location ---
            at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
         --- End of stack trace from previous location ---
            at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 894
            at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.…[truncated]
   2026-09-07 11:58:22Z memex-portal-deployment-74bfcd8f6b-sksm4 fail: Orleans.Messaging[100071]
         Failed to address message Request [S10.244.2.22:11111:147785622 sys.client/hosted-10.244.2.22:11111@147785622]->[ podhub/mesh/fMEHZgBeDE-fQxvBO5Q4oA] MeshWeaver.Connection.Orleans.IPodHubGrain.Attach() #7E91FF6F74043A96
         Orleans.Runtime.OrleansException: Current directory at S10.244.5.41:11111:147772410 is not stable to perform the lookup for grainId podhub/mesh/fMEHZgBeDE-fQxvBO5Q4oA (it maps to S10.244.5.124:11111:147785772, which is not a valid silo). Retry later.
            at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 890
            at Orleans.Runtime.TaskRequest`1.Invoke() in /_/src/Orleans.Core.Abstractions/Runtime/GrainReference.cs:line 714
         --- End of stack trace from previous location ---
            at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Core/Core/GrainMethodInvoker.cs:line 133
            at MeshWeaver.Hosting.Orleans.AccessContextGrainCallFilter.Invoke(IIncomingGrainCallContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.Orleans/AccessContextGrainCallFilter.cs:line 38
            at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Core/Core/GrainMethodInvoker.cs:line 95
            at Orleans.Runtime.InsideRuntimeClient.Invoke(IGrainContext target, Message message) in /_/src/Orleans.Runtime/Core/InsideRuntimeClient.cs:line 318
         --- End of stack trace from previous location ---
            at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
         --- End of stack trace from previous location ---
            at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 894
            at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirecto…[truncated]
   2026-09-10 12:26:14Z memex-portal-deployment-7456f8f447-lgtrr fail: Orleans.Messaging[100071]
         Failed to address message Request [S10.244.2.49:11111:148046995 sys.client/hosted-10.244.2.49:11111@148046995]->[ podhub/portal/anonymous] MeshWeaver.Connection.Orleans.IPodHubGrain.Attach() #68B0D465DFD023A5
         Orleans.Runtime.OrleansException: Current directory at S10.244.4.87:11111:147918862 is not stable to perform the lookup for grainId podhub/portal/anonymous (it maps to S10.244.5.155:11111:148046913, which is not a valid silo). Retry later.
            at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 890
            at Orleans.Runtime.TaskRequest`1.Invoke() in /_/src/Orleans.Core.Abstractions/Runtime/GrainReference.cs:line 714
         --- End of stack trace from previous location ---
            at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Core/Core/GrainMethodInvoker.cs:line 133
            at MeshWeaver.Hosting.Orleans.AccessContextGrainCallFilter.Invoke(IIncomingGrainCallContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.Orleans/AccessContextGrainCallFilter.cs:line 38
            at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Core/Core/GrainMethodInvoker.cs:line 95
            at Orleans.Runtime.InsideRuntimeClient.Invoke(IGrainContext target, Message message) in /_/src/Orleans.Runtime/Core/InsideRuntimeClient.cs:line 318
         --- End of stack trace from previous location ---
            at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
         --- End of stack trace from previous location ---
            at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 894
            at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:…[truncated]
   
8. rbuergi commented on Sep 18, 2026

   

   rbuergi

   on Sep 18, 2026

   Contributor

   More actions

   Re-closing — the classifier fix is on main and pinned by a test; the residual log line is Orleans' own

   Last seen 2026-09-10 12:26:14Z (the Last seen: in the bot's own reopen comment, 2026-09-17T07:52:07Z) — quiet 7.7 days as of 2026-09-18T06:05Z, while ingest is live (incidents were still being updated 2026-09-18T05:42Z).

   Defect item 1 — "are these calls retried?" — is fixed in current origin/main. Read, not taken from the PR title:

   git show origin/main:src/MeshWeaver.Connection.Orleans/OrleansRoutingService.cs | sed -n '790,850p'
   

   IsDirectoryUnstable (:825) matches the bare OrleansException on three markers, and both phrases this issue reported are among them:

   • DirectoryHopLimitMarker = "hop limit is reached" (:848) — whose doc comment names this ticket: "the shape prod logged during two rolling deploys (issue Grain calls are dropped instead of retried when the Orleans directory is unstable during a rollout #2357)";
   • DirectoryRetryLaterMarker = "Retry later" (:842);
   • DirectoryInvalidSiloMarker = "on invalid silo" (:875, added later for the registration side).

   RoutingGrain.cs:1791 reaches it from ClassifyDeliveryException ("🚨 THE BARE OrleansException THE TYPE TEST ABOVE CANNOT SEE — issue #1742 / #2357"), so the existing retry-with-fresh-resolve in RoutingGrain.DeliverToGrainObservable is now actually entered. Pinned by test/MeshWeaver.Hosting.Orleans.Test/OrleansDirectoryInstabilityClassificationTest.cs.

   Why the counter kept moving anyway. I read the live incident record Admin/_LogIncident/830c48fa8c2283d1 (read-only, on the control instance). Every post-close sample (2026-09-03, 2026-09-07 ×5, 2026-09-10) is the "Current directory at S… is not stable … Retry later." shape on IPodHubGrain.Attach() — i.e. precisely the condition the classifier now treats as transient and retries. The line itself is emitted by Orleans' own Orleans.Messaging logger at the addressing layer, below where MeshWeaver can suppress it, so a perfect fix still leaves the log line and the incident counter ticking during a rolling deploy. The defect this issue names — dropped instead of retried — is what was fixed, and it is verified in code.

   Item 2 (the Memory stream provider in a clustered deployment) was separately re-sized on this thread and belongs to the durable-stream-provider work, not here.

   The 2026-09-17T07:51–07:53Z reopen was part of a bulk wave whose predicate read an accumulated counter rather than "occurred since the close" — analysed in Reading a Recurrence Reopen.

   Closing a dormant incident is safe by design: a recurrence re-opens the issue automatically with fresh evidence, so nothing is lost by closing while quiet.
9. rbuergi

   closed this as not plannedon Sep 18, 2026
10. rbuergi

    added a commit that references this issue on Sep 19, 2026

    review(#4923): narrow the departed-silo guard off the OrleansExceptio…

    ...

    7b755ad