RoutingGrain routing and NACK delivery execute after the Autofac root scope is disposed at silo shutdown #2638

Activity

1. rbuergi commented on Aug 29, 2026

   

   rbuergi

   on Aug 29, 2026

   Contributor

   More actions

   PR #2647 — the window now fails clean and correctly classified. It does NOT close the window.

   Your mechanism is confirmed from source, and the diagnosis was right on both counts. Two things landed; one is deliberately left, and it is the one this issue rates at medium confidence.

   Fixed

   1. The classification, which is the functional half. Delivery to 'Planning' failed: Instances cannot be resolved … was NACK'd ErrorType.Failed — terminal — because ClassifyDeliveryException recognised only IsDirectoryUnstable and IsShutdownShaped. Consumers with recovery machinery of their own (SynchronizationStream's resubscribe latch, MeshNodeStreamCache's transient-owner rule) ride out ShuttingDown and tear down on a terminal verdict, so a routine pod exit permanently killed mirrors that would have resumed on the surviving pod. That is precisely the damage #2346/#2357 removed for the other two shapes and left standing for this one.

   It is gated on a probe of the container, not on the bare ObjectDisposedException type test — the same shape MessageHub.IsTerminatedByScopeTeardown uses for #2444. An unrelated disposed dependency is a genuine defect and stays terminal; a negative control pins that.

   2. The dual-transport AggregateException. Exactly as you called it: the stream fallback resolves through the same dead scope, so it could only restate the first fault. It is now skipped when the first transport died on a disposed container, and the outcome is stated once at Information naming the cause. A process finishing its shutdown is not an incident.

   Left open — and it is the root cause, so please read this before closing

   Medium confidence on which exact lifecycle stage disposes the scope while a routing turn is still executing — that needs confirming in the host stop sequence.

   I did not resolve that, and I did not want to guess at it, because the fix it implies is a change to the host stop sequence rather than to the router: routing work would have to be quiesced before the container is disposed. The precedent you cite (MessageService.Post's RunLevel gate) is a hub-level short-circuit; the equivalent here is not obvious, and two candidate shapes are meaningfully different:

   • a gate on RoutingGrain — but the gate OrleansRoutingService.GrainWhileRunning implements is documented as deliberately not applied to the drain: a message already accepted for routing must still land, and Orleans' own placement is what sends it to a healthy silo. Refusing there would drop live work rather than relocate it;
   • an ordering change — a hosted service, or a lifecycle stage, that waits for in-flight routing turns before the root scope is torn down. That is the correct-sounding one and it needs someone who can say authoritatively where in the silo stop sequence the Autofac root goes, and what "in-flight routing turns" is measurable as.

   The precise open question: at which stage of the silo host stop sequence is the Autofac root LifetimeScope disposed relative to RoutingGrain turn quiescence, and is there a lifecycle hook that can hold disposal until the routing pool has drained without deadlocking on it (the IMeshService-constructed-with-the-calling-hub hazard)? Until that is answered, #2647 makes the window bounded, quiet and correctly classified — but the window is still there, and every rolling deploy still re-creates it.

   Not #2635

   Worth recording since the two look adjacent: #2635 / PR #2644 is the test-harness DisposalCompleted join. Different code path, no file overlap, and it does not reach this.
2. rbuergi

   added a commit that references this issue on Aug 29, 2026

   Merge pull request #2647 from Systemorph/fix/2638-routing-after-scope…

   ...

   7828edd
3. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   Contributor

   More actions

   The open question, answered from source — and the seam it names (PR #2726)

   At which stage of the silo host stop sequence is the Autofac root LifetimeScope disposed relative to RoutingGrain turn quiescence, and is there a lifecycle hook that can hold disposal until the routing pool has drained without deadlocking on it?

   The root goes LAST, and the ordering the comments claim is real. The host's root container is the mesh's Autofac container (MeshHostApplicationBuilder.cs:46 → MessageHubServiceProviderFactory → ServiceProviderExtensions.cs:57-60). The root hub never disposes it (MessageHub.cs:2295, deliberately commented out). The host disposes it in RunAsync's using, after StopAsync has fired ApplicationStopping, stopped every hosted service in reverse order — SiloHostedService before MeshTeardownHostedService, Kestrel after both (OrleansServerRegistryExtensions.cs:39-45) — and run StoppedAsync, where MeshTeardownHostedService drains the mesh root hub. Inside the silo stop (Orleans 10.2.2 Silo.Participate, Silo.cs:466-486; LifecycleSubject.OnStop descending, same stage in parallel): Active 20000 → BecomeActive 19999 (membership ShuttingDown) → GrainDeactivation 19998 (Catalog.DeactivateAllActivations, 30 s DeactivationTimeout per activation) → RuntimeServices 4000 (BlockApplicationMessages) → RuntimeInitialize 2000 (message centre stop) → First (IoPoolSiloTeardown join).

   So the window is not "the scope goes too early" — it is "nothing ever waits for a route". Three tiers, none joined:

   1. RoutingGrain.RouteMessage is O(1) and returns Forwarded (One non-reentrant RouteMessage can wedge a whole silo's routing — no timeout on that path #1028); Orleans never sees a request in flight on the routing grain, so DeactivateAllActivations deactivates it instantly and the DeactivationTimeout wait never applies. The "a turn parked in StreamPostTimeout / retries / the subscriber probe outlives the deactivation budget" hypothesis is false — there is no such turn.
   2. The leg runs on the routing pool, but SubscribeThroughPool holds its permit for the subscribe window only (IoPool.cs:391-440); past that, IoPoolSiloTeardown's join sees nothing, and the drain at stage First disposes the leg and completes it silently.
   3. The per-node delivery (DeliverToGrainWithRetry) was subscribed inside BuildGrainRoute's .Select with the subscription discarded, and every PostFailure NACK was a bare .Subscribe — detached from even the leg. Their tails (grain call, ≤6 retry timers on Scheduler.Default, the NACK) ran through grain deactivation, the message-centre stop, the mesh drain and the container disposal. That is exactly the prod trace: Delivery to 'Planning' failed: Instances cannot be resolved… is a retry's GetGrain resolving the codec provider from the disposed scope, then the NACK about it dying the same way.

   The hook. A silo lifecycle participant at ServiceLifecycleStage.Active — the first thing the stop does, before membership announces ShuttingDown and before a single grain deactivates — holding on a mesh-scoped gauge (RoutingQuiescence) that counts every leg and NACK from dispatch to termination. It cannot deadlock: the wait is a subscription bridged once at the boundary (ReactiveCompletion.ObserveCompletion, not .ToTask()), it parks nothing a leg needs (legs run on pool/thread-pool threads, a fully running scheduler answers their grain calls, their NACKs land on hubs that are disposed only in StoppedAsync), and it converges because ApplicationStopping fires before any hosted service stops, so OrleansRoutingService.DeliverMessage already refuses new dispatches — the count is monotone non-increasing from the first stop step. Bounded at 30 s, residual named at Error; a non-graceful stop holds nothing.

   Not a gate — nothing is refused. This is the "ordering" shape you asked for: accepted work lands, over a transport that still exists, and the per-node delivery is now composed into the leg so the pool drain terminates whatever the hold released over. Nothing routing-related executes after the silo has stopped, let alone after the container is gone.

   Pinned by RoutingQuiescenceTest (real SiloLifecycleSubject, a probe at the grain-deactivation stage; mutation-verified — the pre-fix shape fails at "the silo stop must be HELD … but found True") and RoutingQuiescenceClusterTest (a stalled leg is counted on the silo's gauge until it terminates).
4. rbuergi

   added a commit that references this issue on Aug 30, 2026

   fix(#2638): hold the silo stop until accepted routing work has landed (…

   ...

   81f8869
5. rbuergi

   closed this as completedin #2726on Aug 30, 2026
6. rbuergi

   added a commit that references this issue on Aug 30, 2026

   fix(#2679): the permission fold resolves its services once, and a ren…

   ...

   a9a06e3
7. systemorph-com

   reopened this on Sep 17, 2026
8. systemorph-com commented on Sep 17, 2026

   

   systemorph-combot

   on Sep 17, 2026

   Contributor

   More actions

   Reopened — this fault is happening again.

   Still occurring: 5 more since the last update (7 total).

   • Last seen: 2026-08-29 06:37:19Z
   • Pods: memex-portal-deployment-7ccdd994cf-sbrjg, memex-portal-deployment-7c6b48957d-xml97

   Recent log lines

   2026-08-29 06:37:18Z memex-portal-deployment-7c6b48957d-xml97 fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
         [ROUTE] Cannot deliver the Failed DeliveryFailure for 5Y4D6OIn1kyo4s3WmCLiAg to sender portal/nodeops-Uhl2cThiZEO9fsFEXx5l3Q: BOTH transports failed — the directed call to its grain/pod-hub activation AND the stream publish. The sender will wait out its own request budget — the original failure was: Delivery to 'AgenticPrimerDe' failed: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.. 0 earlier undeliverable NACK(s) to this sender since the last such line were logged at Debug.
         System.AggregateException: Neither the directed pod-hub call nor the stream publish could carry the NACK. (Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.) (Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.)
          ---> System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
            at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
            at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
            at Orleans.Serialization.GeneratedCodeHelpers.OrleansGeneratedCodeHelper.GetService[TService](Object caller, ICodecProvider codecProvider) in /_/src/Orleans.Serialization/GeneratedCodeHelpers/OrleansGeneratedCodeHelper.cs:line 58
            at Proxy_IPodHubGrain(Object, GrainReferenceShared, IdSpan)
            at Orleans.GrainFactory.GetGrain[TGrainInterface](String primaryKey, String grainClassNamePrefix) in /_/src/Orleans.Core/Core/GrainFactory.cs:line 61
            at MeshWeaver.Hosting.Orleans.RoutingGrain.<>c__DisplayClass30_0.<PostFailure>b…[truncated]
   2026-08-29 06:37:19Z memex-portal-deployment-7c6b48957d-xml97 fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
         [ROUTE] Cannot deliver the Failed DeliveryFailure for wbh2bB-Yxkam9IUOG_zueQ to sender portal/nodeops-Uhl2cThiZEO9fsFEXx5l3Q: BOTH transports failed — the directed call to its grain/pod-hub activation AND the stream publish. The sender will wait out its own request budget — the original failure was: Delivery to 'AdvancedBusinessRules' failed: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.. 0 earlier undeliverable NACK(s) to this sender since the last such line were logged at Debug.
         System.AggregateException: Neither the directed pod-hub call nor the stream publish could carry the NACK. (Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.) (Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.)
          ---> System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
            at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
            at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
            at Orleans.Serialization.GeneratedCodeHelpers.OrleansGeneratedCodeHelper.GetService[TService](Object caller, ICodecProvider codecProvider) in /_/src/Orleans.Serialization/GeneratedCodeHelpers/OrleansGeneratedCodeHelper.cs:line 58
            at Proxy_IPodHubGrain(Object, GrainReferenceShared, IdSpan)
            at Orleans.GrainFactory.GetGrain[TGrainInterface](String primaryKey, String grainClassNamePrefix) in /_/src/Orleans.Core/Core/GrainFactory.cs:line 61
            at MeshWeaver.Hosting.Orleans.RoutingGrain.<>c__DisplayClass30_0.<PostFai…[truncated]
   2026-08-29 06:37:19Z memex-portal-deployment-7c6b48957d-xml97 fail: MeshWeaver.Hosting.Orleans.RoutingGrain[0]
         [ROUTE] Cannot deliver the Failed DeliveryFailure for STj1s0H100ORJ4t98MEnAA to sender cache/Uhl2cThiZEO9fsFEXx5l3Q: BOTH transports failed — the directed call to its grain/pod-hub activation AND the stream publish. The sender will wait out its own request budget — the original failure was: Delivery to 'AgenticPrimer' failed: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.. 0 earlier undeliverable NACK(s) to this sender since the last such line were logged at Debug.
         System.AggregateException: Neither the directed pod-hub call nor the stream publish could carry the NACK. (Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.) (Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.)
          ---> System.ObjectDisposedException: Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.
            at Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException()
            at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
            at Orleans.Serialization.GeneratedCodeHelpers.OrleansGeneratedCodeHelper.GetService[TService](Object caller, ICodecProvider codecProvider) in /_/src/Orleans.Serialization/GeneratedCodeHelpers/OrleansGeneratedCodeHelper.cs:line 58
            at Proxy_IPodHubGrain(Object, GrainReferenceShared, IdSpan)
            at Orleans.GrainFactory.GetGrain[TGrainInterface](String primaryKey, String grainClassNamePrefix) in /_/src/Orleans.Core/Core/GrainFactory.cs:line 61
            at MeshWeaver.Hosting.Orleans.RoutingGrain.<>c__DisplayClass30_0.<PostFailure>b__2() in /h…[truncated]
   
9. rbuergi commented on Sep 17, 2026

   

   rbuergi

   on Sep 17, 2026

   Contributor

   More actions

   Re-closed — the reopen above was on stale evidence (#4577).

   The recurrence bot reopened this at 2026-09-17T07:52:57Z under the headline "this fault is happening again", but the occurrence its own comment cites is OLDER than the close it reverted:

   the bot's Last seen:	the close it reverted
   2026-08-29 06:37:19Z	2026-08-30T07:14:35Z

   So the counter moved, not the clock — the fault's newest occurrence predates the close, which is the one thing "still happening" has to mean. Root cause and fix: LogIncidentFiler.Reopen reopened on the sole condition that the issue was closed, while the comment-debt clock pays a final count for incidents whose fault has stopped (the fix, Systemorph/MeshWeaver.Plugins#2014, is written but NOT yet merged — it makes a reopen require incident.LastSeen > issue.ClosedAt).

   This restores the close exactly as it stood; nothing is asserted about the defect itself. If the fault happens again, the next occurrence postdates this close and reopens the issue on evidence that means what it says.

   _{One of 53 issues whose reopen was measured stale (129 bot reopens on 2026-09-17, 125 of them inside the 07:51-07:53Z sweep; 76 postdate their close and were left open). Measured and re-closed 2026-09-17 by Claude Opus 5.}
10. rbuergi

    closed this as completedon Sep 17, 2026