fix(#2638): hold the silo stop until accepted routing work has landed

[rbuergi]

Closes #2638. Residual of #2647, which made the window fail clean and classified and left the root cause open.

The open question, answered from source

At which stage of the silo host stop sequence is the Autofac root LifetimeScope disposed relative to RoutingGrain turn quiescence, and is there a lifecycle hook that can hold disposal until the routing pool has drained without deadlocking on it?

Where the root goes. The host's root container IS the mesh's Autofac container (MeshHostApplicationBuilder → Host.ConfigureContainer(new MessageHubServiceProviderFactory(BuildHub)), src/MeshWeaver.Hosting/MeshHostApplicationBuilder.cs:46; MessageHubServiceProviderFactory.CreateServiceProvider → CreateMeshWeaverServiceProvider → new AutofacServiceProvider(containerBuilder.Build()), src/MeshWeaver.ServiceProvider/ServiceProviderExtensions.cs:57-60). The root hub does not dispose it — MessageHub.cs:2295 has that line commented out on purpose. It is disposed by the host, in RunAsync's using, strictly after StopAsync has (1) fired ApplicationStopping, (2) stopped every IHostedService in reverse registration order — SiloHostedService among them (UseOrleans runs after the MeshHostApplicationBuilder ctor in UseOrleansMeshServer, OrleansServerRegistryExtensions.cs:39-45, so the silo stops before MeshTeardownHostedService and Kestrel stops after both) — and (3) run IHostedLifecycleService.StoppedAsync, where MeshTeardownHostedService drains the mesh root hub (MeshTeardownHostedService.cs:72-108).

What the silo stop does, in order (Orleans 10.2.2, Silo.Participate, Silo.cs:466-486; stops run in descending stage, same-stage observers in parallel, LifecycleSubject.OnStop, LifecycleSubject.cs:154-186):

stage	value	stop step
Active	20000	OnActiveStop: gateway stop-send, grain services
BecomeActive	19999	MembershipAgent.OnBecomeActiveStop → membership ShuttingDown
GrainDeactivation	19998	Catalog.DeactivateAllActivations(ct) — each activation bounded by GrainCollectionOptions.DeactivationTimeout (30 s)
RuntimeServices	4000	messageCenter.BlockApplicationMessages()
RuntimeInitialize	2000	messageCenter.StopAsync(); Terminated
First	int.MinValue	IoPoolSiloTeardown: IoPoolRegistry.Dispose() + join

Why none of it waited for a route. Three tiers, none joined:

1. The turn. RoutingGrain.RouteMessage does O(1) work and returns Forwarded (One non-reentrant RouteMessage can wedge a whole silo's routing — no timeout on that path #1028) — the leg is handed to the routing pool. From Orleans' point of view the routing grain never has a request in flight, so DeactivateAllActivations deactivates it instantly and the DeactivationTimeout wait never applies. The hypothesis "a turn parked in StreamPostTimeout / retries / the subscriber probe outlives the deactivation budget" is false — there is no such turn.
2. The leg. SubscribeThroughPool holds its permit for the subscribe window only (IoPool.cs:391-440, and MeshTeardownExtensions.cs:60-66 says so). A leg past its subscribe is invisible to IoPoolSiloTeardown's join; the drain at stage First disposes it and completes it silently — no NACK.
3. The delivery and the NACK — detached from even that. BuildGrainRoute subscribed DeliverToGrainWithRetry inside a .Select and discarded the subscription; every PostFailure NACK was a bare .Subscribe. So the grain call, its ≤6 retry timers (Scheduler.Default) and its NACK ran on continuations nothing counted and nothing drained. That is the prod tail: Delivery to 'Planning' failed: Instances cannot be resolved… is the retry chain's grainCall() → GetGrain resolving a codec provider from the disposed scope, and the NACK about it dying the same way.

So: the root is disposed after the silo has fully stopped and the mesh has drained — the ordering is what the comments claimed — but routing quiescence was never a stage of anything. The window is not "the scope goes too early"; it is "nothing ever waits for a route".

The hook, and why it cannot deadlock. A silo lifecycle participant at ServiceLifecycleStage.Active — the first thing the stop does, before membership announces ShuttingDown and before a single grain deactivates. The wait is a subscription on a mesh-scoped gauge, bridged once at the boundary with ReactiveCompletion.ObserveCompletion (never .ToTask(), which would resume LifecycleSubject.OnStop inline on the gauge's event-loop thread). It parks nothing a leg needs: legs run on pool/thread-pool threads, their grain calls are answered by a fully running scheduler, their NACKs land on hubs that are still alive (the pod-process hubs are disposed only in StoppedAsync). And the count converges because ApplicationStopping fires before any hosted service stops, so OrleansRoutingService.DeliverMessage already refuses new dispatches as ShuttingDown — the gauge is monotone non-increasing from the first stop step.

The change

• RoutingQuiescence (mesh-scoped, ActivityTracker's reactive shape — deltas in, count out, serialised on one scheduler, no lock/gate) counts every leg and every NACK from dispatch to termination.
• RoutingQuiescenceSiloParticipant holds the silo stop at Active until the count is zero. Bounded at 30 s (the same family as MeshTeardownHostedService / IoPoolSiloTeardown); expiry is reported at Error naming the residual — that leg is the defect, and this line is the only attribution its tail will get. A non-graceful stop (token already cancelled — Silo.Dispose()'s path, on which Orleans' own Active-stage stop returns immediately) holds nothing; the host's ShutdownTimeout expiring mid-hold releases it.
• RoutingGrain: the per-node delivery is part of the leg now (SelectMany, DeliverToGrainRoute returning IObservable<Unit>; DeliverToGrainWithRetry is that leg subscribed, kept for the pure tests). Every NACK goes through the routing pool and the gauge. On a healthy roll the count reaches zero in milliseconds; a leg the hold released over is terminated by the pool drain at stage First, so nothing routing-related executes after the silo has stopped, let alone after the container is gone.

Not a gate: nothing is refused. This is the "ordering" shape the #2647 comment asked for — accepted work lands, over a transport that still exists.

Tests

RoutingQuiescenceTest — real SiloLifecycleSubject (the type Silo.StopAsync awaits), a probe observer at GrainDeactivation recording the count it found:

test	asserts
SiloStop_HoldsBeforeGrainDeactivation_UntilTheRoutingWorkHasLanded	with a leg in flight the stop Task is provably incomplete and the probe has not run; once the leg lands the probe runs and finds 0
SiloStop_WithNothingInFlight_DoesNotHold	control: idle stop, no wait, says so
SiloStop_BudgetExpiry_ReleasesTheSilo_AndNamesTheResidual	bounded, residual at Error
SiloStop_NonGraceful_DoesNotHold_AndReportsWhatItAbandons	cancelled token: no hold
SiloStop_HostBudgetExpiringMidHold_ReleasesTheSilo	host budget cancels the hold
TheGrainDelivery_IsPartOfTheLeg_WhichTerminatesOnlyOnceTheCallHasAnswered	the leg stays in flight while the grain call is pending
TheGrainDelivery_NacksInsideTheLeg_ThenTerminates	NACK arm inside the leg

RoutingQuiescenceClusterTest — real silo, the StallingPathResolver vehicle from #1028: a dispatched leg is counted on the silo's gauge until it terminates, and the silo lifecycle carries the participant.

Non-vacuity, by mutation — participant's OnStop forced to return Task.CompletedTask (the pre-fix shape):

[FAIL] SiloStop_HoldsBeforeGrainDeactivation_UntilTheRoutingWorkHasLanded
   Expected value to be false because the silo stop must be HELD at stage Active while a route leg is in flight …, but found True.
Failed!  - Failed: 5, Passed: 2, Total: 7

Reverted: Passed! - Failed: 0, Passed: 75 across this file plus RoutingGrainDeliveryRetryTest, RoutingAfterScopeTeardownTest, RoutingGrainFailureClassificationTest, RoutingGrainPodHubDeliveryRetryTest, RoutedFailureClassificationTest, RoutingGrainStreamPostBoundTest, OrleansDirectoryInstabilityClassificationTest, SiloShutdownActivationGateTest, OrleansRoutingShutdownClassificationTest, IoPoolSiloTeardownTest, RoutingGrainTurnIsolationTest, StreamPostTimeoutAttributionTest, DeadTargetRefusalWindowTest, RoutingFailureTakesTheLocalRouteTest; and the whole MeshWeaver.Hosting.Orleans.Test project locally (Release).

What's New: 2026-08-30-a-departing-pod-lets-accepted-messages-land.md (Category: Fix).

🤖 Generated with Claude Code

https://claude.ai/code/session_01EPCWramjFYquX5SaMdeJKW

[Copilot]

Pull request overview

This PR addresses #2638 by ensuring a silo shutdown does not proceed past the Orleans Active stop stage while there is still accepted routing work (route legs and NACK delivery) in flight, preventing routing/NACK tails from executing after the host DI container has been disposed.

Changes:

• Introduces a mesh-scoped RoutingQuiescence gauge and an Orleans lifecycle participant that holds silo stop at ServiceLifecycleStage.Active until routing work drains (bounded, with explicit logging on expiry/cancellation).
• Refactors RoutingGrain so grain delivery is composed into the route leg (SelectMany + DeliverToGrainRoute) and NACK delivery is tracked/drained via the routing pool and the quiescence gauge.
• Adds targeted unit + cluster tests validating the participant behavior and real-silo wiring, plus a user-facing “What’s New” entry.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
test/MeshWeaver.Hosting.Orleans.Test/RoutingQuiescenceTest.cs	Adds pure lifecycle/participant tests proving the stop hold, bounds, and delivery-leg composition.
test/MeshWeaver.Hosting.Orleans.Test/RoutingQuiescenceClusterTest.cs	Verifies real-silo DI wiring and that dispatched legs are reflected on the silo’s quiescence gauge until termination.
src/MeshWeaver.Hosting.Orleans/RoutingQuiescence.cs	Adds the routing in-flight gauge, the silo lifecycle participant hold, and DI registration extension.
src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs	Tracks legs/NACKs via RoutingQuiescence, composes grain delivery into the leg, and routes NACK subscriptions through the routing pool for drain visibility.
src/MeshWeaver.Hosting.Orleans/OrleansServerRegistryExtensions.cs	Registers routing quiescence in the Orleans server service wiring.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-30-a-departing-pod-lets-accepted-messages-land.md	Adds a “What’s New” fix entry describing the user-visible effect during rolling deploy shutdown windows.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Test Results (shard 3)

701 tests   509 ✅  4m 57s ⏱️
  5 suites  192 💤
  5 files      0 ❌

Results for commit bf3b1ae.

[github-actions]

Test Results (shard 0)

913 tests   913 ✅  5m 35s ⏱️
  6 suites    0 💤
  6 files      0 ❌

Results for commit bf3b1ae.

[github-actions]

Test Results (shard 5)

1 532 tests   1 532 ✅  5m 25s ⏱️
    8 suites      0 💤
    8 files        0 ❌

Results for commit bf3b1ae.

[github-actions]

Test Results (shard 2)

941 tests   941 ✅  5m 4s ⏱️
  6 suites    0 💤
  6 files      0 ❌

Results for commit bf3b1ae.

[github-actions]

Test Results (shard 4)

   10 files     10 suites   5m 52s ⏱️
3 468 tests 3 468 ✅ 0 💤 0 ❌
4 107 runs  4 107 ✅ 0 💤 0 ❌

Results for commit bf3b1ae.

[github-actions]

Test Results (shard 1)

818 tests   814 ✅  7m 32s ⏱️
  6 suites    4 💤
  6 files      0 ❌

Results for commit bf3b1ae.

[github-actions]

Test Results

   41 files     41 suites   34m 27s ⏱️
8 373 tests 8 177 ✅ 196 💤 0 ❌
9 012 runs  8 816 ✅ 196 💤 0 ❌

Results for commit bf3b1ae.