fix(#2638): routing that runs after the Autofac root scope is disposed fails as a lifecycle transition, not a terminal defect

[rbuergi]

Linking, not closing: #2638.

The mechanism

Orleans builds a grain proxy by resolving its codec provider out of the DI container (OrleansGeneratedCodeHelper.GetService → AutofacServiceProvider.GetService). Once the silo host has disposed its root Autofac LifetimeScope, every remaining delivery — and every NACK about one — faults with Autofac's ObjectDisposedException before it reaches a transport at all.

Prod (memex, 2026-08-29) therefore logged, at Error, an AggregateException carrying the same fault twice:

[ROUTE] Cannot deliver the Failed DeliveryFailure for … to sender cache/…: BOTH transports failed …
        the original failure was: Delivery to 'Planning' failed: Instances cannot be resolved and
        nested lifetimes cannot be created from this LifetimeScope …

Two defects, both fixed

1. The classification was terminal. Delivery to 'Planning' failed: Instances cannot be resolved … was NACK'd as ErrorType.Failed. Consumers with recovery machinery of their own — SynchronizationStream's resubscribe latch, MeshNodeStreamCache's transient-owner rule — ride out ErrorType.ShuttingDown and tear down on a terminal verdict. So a routine pod exit permanently killed live mirrors that would have resumed against the surviving pod seconds later. That is the identical damage #2346/#2357 removed for the directory-unstable and silo-departing shapes, and left standing for this one.

ClassifyDeliveryException now recognises it — gated on a probe of the container itself, never on the bare ObjectDisposedException type test:

internal static bool IsScopeTeardown(Exception ex, Func<bool>? scopeDisposed) =>
    scopeDisposed is not null
    && ExceptionChain.Contains<ObjectDisposedException>(ex)
    && scopeDisposed();

The probe is the same shape MessageHub.IsTerminatedByScopeTeardown uses for #2444, and it is what keeps an unrelated disposed dependency — a genuine defect — terminal. With no probe supplied the answer is exactly the pre-fix one. The walk is ExceptionChain's graph walk, because this arrives through Rx Catch arms and a two-transport AggregateException where which fault sits at index 0 is a race.

2. The second transport was attempted on the same dead container. PostFailure's stream fallback resolves through the very scope the directed call just died on, so it could only restate the first fault. When the first transport dies on a disposed container the fallback is now skipped, and the outcome is stated once, at Information, naming the cause — a process finishing its shutdown is not an incident to page on. The refusal is still logged, still windowed by the same nackLog, and still tells the reader the sender will wait out its budget.

The sender still waits out that budget — nothing running inside a dead container can prevent that — but it is now a bounded wait against a stated cause, and the verdict it eventually reads is one its recovery machinery can act on.

Tests — RoutingAfterScopeTeardownTest

Pure: the classifier and the retry primitive are internal static, the exception text is verbatim from the production log, the container probe is a delegate. No cluster, no host, no clock.

test	asserts
DisposedContainer_ClassifiesAsShuttingDown	the verdict flips from terminal to ride-it-out
DisposedContainer_IsSeenThroughAnAggregate	the prod NACK shape (one fault, twice) is seen regardless of index
DisposedDependency_WhileTheContainerLives_StaysTerminal	negative control — the probe is what makes this a teardown statement; without it, and for a live container, the verdict stays Failed
TheOtherClassificationsAreUnchanged	#2357's SiloUnavailableException still ShuttingDown; a real defect during shutdown still Failed
DeliveryOnADeadContainer_NacksTheSenderAsShuttingDown_WithoutRetrying	end to end through DeliverToGrainWithRetry: one attempt (a dead container is not transient), NACK carries ShuttingDown

Non-vacuity, by mutation

The tests cannot be compiled against unfixed code — the fix adds the probe parameter they pass — so non-vacuity is proven the way #2635 asks for: IsScopeTeardown forced to the pre-fix answer (false &&), which is exactly what the classifier could see before.

[FAIL] DisposedContainer_IsSeenThroughAnAggregate
   Expected value to be ShuttingDown … but found Failed.
[FAIL] DisposedContainer_ClassifiesAsShuttingDown
   Expected value to be ShuttingDown … but found Failed.
[FAIL] DeliveryOnADeadContainer_NacksTheSenderAsShuttingDown_WithoutRetrying
   Expected value to be ShuttingDown … but found Failed.
Failed!  - Failed: 3, Passed: 2, Total: 5

The two that still pass are the negative controls — correctly, since they assert the unchanged behaviour.

With the mutation reverted, this file plus every neighbouring routing/classification suite:

Passed!  - Failed: 0, Passed: 52, Total: 52

(RoutingAfterScopeTeardownTest, RoutingGrainFailureClassificationTest, OrleansDirectoryInstabilityClassificationTest, RoutedFailureClassificationTest, RoutingGrainDeliveryRetryTest, RoutingGrainPodHubDeliveryRetryTest, RoutingFailureTakesTheLocalRouteTest, DeadTargetRefusalWindowTest, RoutingGrainStreamPostBoundTest.)

What this deliberately does NOT do

The issue's own root-cause framing — "the silo shutdown sequence disposes the service provider before routing work is quiesced" — is carried at medium confidence, and quiescing routing before container disposal is a change to the host stop sequence, not to the router. This PR makes the window fail clean and correctly classified; it does not close the window. See my comment on #2638 for the precise open question.

Not related to #2644

#2644 (issue #2635) joins DisposalCompleted in test harnesses; this is the production silo-shutdown path. No file overlap.

🤖 Generated with Claude Code

[rbuergi]

On the test method — what is and is not pinned

Being explicit, because "a test that passes because the race did not happen is worthless":

Pinned deterministically. Everything downstream of the race — the classification of a disposed-container fault, that it is seen through an AggregateException regardless of index, that a disposed dependency on a live container stays terminal, that the delivery leg does not burn its retry budget on a dead container, and that the NACK's second transport is skipped. The container state is a Func<bool> and the exception text is verbatim from the log, so none of it depends on timing.

NOT pinned, and I am not claiming otherwise. The race itself — a RoutingGrain turn still executing when the silo host disposes the Autofac root. Reproducing that needs a real silo torn down at a controlled instant relative to an in-flight routing turn, and a test that spun a cluster and hoped to hit the window would pass most of the time whether the fix were present or not. So I did not write one. What these tests assert is: given the window is entered, the outcome is now a correctly-classified, once-stated lifecycle failure instead of a terminal verdict and a dual-fault AggregateException.

Closing the window itself is the open design question in my comment on #2638, and a test for it belongs with that change — it would be a host-stop-sequence ordering assertion, not a routing one.

[Copilot]

Pull request overview

This PR addresses #2638 by making Orleans routing failures that occur after the Autofac root scope is disposed during silo shutdown behave like a lifecycle transition (classified as ErrorType.ShuttingDown) rather than a terminal defect, and by avoiding redundant “second transport” attempts that can only fail on the same disposed container.

Changes:

• Adds a “container disposed” probe and uses it to classify ObjectDisposedException-shaped routing failures as ShuttingDown (opt-in via probe so unrelated disposed dependencies remain terminal).
• Downgrades logging for shutdown-shaped directed delivery failures and skips the stream-publish fallback when the container is confirmed disposed (to avoid duplicate-fault AggregateException noise).
• Introduces a focused test suite (RoutingAfterScopeTeardownTest) covering classification through aggregates, negative controls, and end-to-end retry/NACK behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs	Adds a DI-scope disposal probe, extends failure classification with scope-teardown detection, adjusts log severity on shutdown-shaped failures, and avoids attempting a second transport when the container is already disposed.
test/MeshWeaver.Hosting.Orleans.Test/RoutingAfterScopeTeardownTest.cs	New unit tests pinning the expected ShuttingDown classification and non-retry behavior for disposed-container routing/NACK scenarios, including negative controls.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[rbuergi]

Not correct — measured rather than argued, because the overload set here is genuinely confusing.

ObjectDisposedException has three constructors and only two of them touch ObjectName:

public ObjectDisposedException(string? objectName)                      // Message = SR.ObjectDisposed_Generic
public ObjectDisposedException(string? objectName, string? message)     // sets _objectName
public ObjectDisposedException(string? message, Exception? innerException) : base(message, innerException)

ScopeDisposed() binds the third — (string message, Exception? innerException) — which passes the text straight to Exception(message, inner) and leaves _objectName empty. The Message override only appends the "Object name: 'x'" line when ObjectName is non-empty, so with it empty the property returns base.Message unchanged.

Verified against net10.0:

ObjectName=[]
Message=[Instances cannot be resolved and nested lifetimes cannot be created from this LifetimeScope as it (or one of its parent scopes) has already been disposed.]
MATCHES_VERBATIM=True

So the setup is consistent with its documentation: the exception carries the production text verbatim, and a future assertion on that exact message would hold.

Worth adding for completeness, since it is the more important half: no assertion in this PR depends on the message at all, and that is deliberate. IsScopeTeardown classifies on ExceptionChain.Contains<ObjectDisposedException> plus the container probe — never on text — precisely because Autofac's wording is not a contract we control. The verbatim text is there so a reader can match the test to the incident log, not so the classifier can match on it.

[github-actions]

Test Results (shard 3)

495 tests   491 ✅  4m 43s ⏱️
  6 suites    4 💤
  6 files      0 ❌

Results for commit 11b885c.

[github-actions]

Test Results (shard 2)

957 tests   957 ✅  5m 9s ⏱️
  6 suites    0 💤
  6 files      0 ❌

Results for commit 11b885c.

[github-actions]

Test Results (shard 4)

   10 files     10 suites   5m 5s ⏱️
3 773 tests 3 581 ✅ 192 💤 0 ❌
4 405 runs  4 213 ✅ 192 💤 0 ❌

Results for commit 11b885c.

[github-actions]

Test Results (shard 0)

907 tests  ±0   907 ✅ ±0   5m 33s ⏱️ -1s
  6 suites ±0     0 💤 ±0 
  6 files   ±0     0 ❌ ±0 

Results for commit 11b885c. ± Comparison against base commit ecef6db.

[github-actions]

Test Results (shard 5)

1 585 tests  ±0   1 584 ✅ ±0   7m 18s ⏱️ +14s
    8 suites ±0       1 💤 ±0 
    8 files   ±0       0 ❌ ±0 

Results for commit 11b885c. ± Comparison against base commit ecef6db.

[github-actions]

Test Results (shard 1)

1 020 tests   1 020 ✅  6m 25s ⏱️
    6 suites      0 💤
    6 files        0 ❌

Results for commit 11b885c.

[github-actions]

Test Results

   42 files     42 suites   34m 16s ⏱️
8 737 tests 8 540 ✅ 197 💤 0 ❌
9 369 runs  9 172 ✅ 197 💤 0 ❌

Results for commit 11b885c.