fix(routing): a departed silo is a lifecycle transition, not a delivery defect (#2299 departed-silo half; #2307 consolidated) by rbuergi · Pull Request #4923 · Systemorph/MeshWeaver

rbuergi · 2026-09-19T19:13:24Z

Addresses the departed-silo half of the delivery classifier, which is the half of #2299 that PR #4914 explicitly leaves out ("I also only cover the shape I have ground truth for. The triage mentions 'two shapes'; the incident's newest samples are all this one"). The two PRs touch different predicates and do not conflict: #4914 adds IsDeactivatedActivation beside ClassifyDeliveryException; this one extends IsShutdownShaped at the file's tail.

The gap

ClassifyDeliveryException answers ShuttingDown for IsDirectoryUnstable ‖ IsShutdownShaped ‖ IsScopeTeardown, and the host half of that — IsShutdownShaped — was two exception TYPES: SiloUnavailableException, OrleansLifecycleCanceledException. The two ways Orleans actually reports a departed pod are neither, so after the six re-resolving retries the sender got a terminal Failed, which the production line says verbatim: "surfacing Failed DeliveryFailure to sender Hosting/Deployment".

The cost is not the message, it is the subscription: SynchronizationStream's resubscribe latch and MeshNodeStreamCache's transient-owner rule ride out ShuttingDown and tear down on Failed. One ordinary roll therefore killed live views and mirrors whose target was answering again on the surviving pod seconds later.

Exception while sending message: …ConnectionFailedException: Unable to connect to
S10.244.4.87:11111:146498551, will retry after 585.8766ms

The target silo is no longer active: target was S10.244.4.183:11111:146524552,
but this silo is S10.244.4.183:11111:146534005

Why these clear the classifier's own bar

The rule is only a lifecycle transition by construction, and "anything this does not recognise stays terminal, so a genuine defect is still reported as one". A SiloAddress is generation-stamped (S<ip>:<port>:<generation>), so each shape is a statement about ONE silo incarnation — one whose socket refuses connections, or one explicitly superseded by a named successor. Neither can answer at that address again, and the only cure is the re-resolve the retry already performs.

Nothing new spins. IsTransientFailure's OrleansMessageRejectionException type test has matched both shapes since #2314, so they have been retried with a fresh resolve six times all along. The defect was a classifier that could not read its input — the same shape as #1742 / #2357 / #3139 / #2451. A bare TimeoutException deliberately stays terminal: the silo accepted the connection and did not answer, i.e. plausibly wedged, and demoting that is a resubscribe storm.

And what the verdict arms is bounded, which is the objection this file raises against a generous rule: MeshNodeStreamCache's transient-fault breaker gives a transient claim three grace failures and then backs re-probes off exponentially (1 s → 60 s cap), on the stated reasoning that a streak is empirical proof the transient claim was false.

🚨 Two arms, and neither is the `OrleansException` base

The first revision guarded both phrases on the OrleansException base, mirroring IsDirectoryUnstable. Review caught that this is broader than the signal, correctly, and the fix went further than narrowing — because narrowing alone would have lost coverage in the one direction this file has an incident about.

Arm 1 — a TYPE, no prose. Orleans.Runtime.Messaging.ConnectionFailedException is public (probed against the pinned 10.3.1) and is thrown only by ConnectionManager.GetConnectionAsync(SiloAddress), i.e. only ever about a cluster endpoint. Stronger than any phrase, and it cannot go inert when Orleans re-words a message. It is also the arm that catches Orleans' carried exception wins resolution (rejection?.Exception ?? new OrleansMessageRejectionException(…)), where the caller receives the connect failure bare with no rejection wrapper in the graph — the mechanism #1742 / #2357 exist for. Narrowing to the rejection type and stopping there would have re-opened that hole for this shape.

Arm 2 — the phrase, guarded by the CONCRETE rejection. The superseded-generation shape needs it: Orleans rejects that one with no carried exception, so the caller gets OrleansMessageRejectionException and its detail exists only as text. The connect shape is covered twice over, because production's wrapper embeds the inner's text.

Net effect on the review's case: a bare application-level OrleansException quoting these words now stays terminal.

There is no typed rejection reason to read, and OrleansMessageRejectionException carries every refusal kind, so even that type alone is not enough — hence arm 2 keeps the phrase. Both phrases are pinned against the SHIPPED Orleans assemblies' own string literals: Unable to connect to is a literal of Orleans.Core, The target silo is no longer active of Orleans.Runtime. An Orleans re-wording reds the pin instead of letting the classifier go silently inert, and if it does re-word, this returns to the pre-fix answer rather than misclassifying anything.

IsShutdownShaped also now walks the exception graph rather than the InnerException line, for the reason IsScopeTeardown already gives — which fault sits at index 0 of PostFailure's two-transport AggregateException is a race. That narrowness applied to the two pre-existing type tests as well.

Verification

dotnet build -c Release -warnaserror, one project per invocation: MeshWeaver.Hosting.Orleans, MeshWeaver.Documentation, MeshWeaver.Hosting.Orleans.Test, MeshWeaver.Documentation.Test — 0 Warning(s), 0 Error(s) each, none of them an up-to-date no-op. Suites: MeshWeaver.Hosting.Orleans.Test 278/278, MeshWeaver.Documentation.Test 601/601 — the latter hosts the async / static / observable-bridge guards and the topic-map guard that caught a missing Architecture.md entry.

Mutation-measured five ways, because over-accepting here masks defects. These are the numbers that were measured, not predicted — one prediction I wrote on the review thread was wrong and is corrected there:

mutation	red
predicate not wired into `IsShutdownShaped` (the pre-fix answer)	6 — the 4 verbatim production shapes, `AConnectionFailure_IsAcceptedByTypeWithoutAnyPhrase`, `ItIsFoundBesideAnotherFaultInAnAggregate_NotOnlyAtIndexZero`
arm 2 (the phrase) dropped	5 — the 4 production shapes + the aggregate fact
arm 1 (the type) dropped	1 — `AConnectionFailure_IsAcceptedByTypeWithoutAnyPhrase`
arm 2's guard widened back to `OrleansException`	1 — `ABareOrleansExceptionCarryingThePhrase_StaysTerminal` (the review's case)
both guards dropped, phrase matched on any exception	3 — adds `AnApplicationConnectFailureCarryingTheSamePhrase_StaysTerminal`

Every exception text in the test is quoted verbatim from the production log rather than paraphrased, and the production shapes are built through the real OrleansMessageRejectionException — its constructors are internal, so via Activator with NonPublic binding, with a self-check that the instance carries the message, because a base-class stand-in would have made every fact a statement about OrleansException instead.

#2299 and #2307 are ONE root, and the consolidation

They were filed hours apart and read as two defects for four weeks. Same condition, opposite ends of the same call:

	#2299	#2307
fingerprinted on	`MeshWeaver.Hosting.Orleans.RoutingGrain` — our line	`Orleans.Messaging` `100071` — Orleans' line
counts	one per verdict	one per attempt
occurrences	191	3,959

Both legs are composed by RoutingGrain, both retried by the same DeliverToGrainObservable, both classified by the same ClassifyDeliveryException — and the two issues already share a fix branch in their history (fix/2299-2307-pod-hub-delivery-retry, PR #2314, which landed the retry half for both at once).

#2307's stated defect — "something above IMessageHubGrain is re-sending on a fixed short interval against a cached, dead address without re-resolving placement or backing off" — does not exist in src/: both retry sites are Observable.Defer-cold (every attempt re-invokes GetGrain, i.e. re-resolves placement), bounded (6 and 5), and exponentially backed off (250 ms → 3 s). Its "distinct message ids ~1 Hz apart" is what one delivery's retry ladder looks like from inside Orleans. #2307 has therefore been consolidated into #2299 as the surviving ticket.

What this does NOT do

#2299 stays open, on the shape neither this PR nor #4914 touches: the self-sustaining re-activation bounce. PodHubGrain.Deliver on a silo with no local route calls DeactivateOnIdle() and throws PodHubNotHereException, [PreferLocalPlacement] puts the next attempt on the caller again, and messages that reach the deactivating activation come back as Orleans' forwarding rejection. That needs the deactivation/re-activation interplay traced, not a classifier edit. This change alters what the sender is told; it does not stop that loop.

I also did not touch IsTransientFailure. A bare ConnectionFailedException (the carried-exception-wins path) would not be retried by its type test, but every production sample has OrleansMessageRejectionException at the top, so that is a hypothesis with no evidence behind it and it is mirrored in two files — not something to change blind.

Findings committed as ADepartedSiloIsNotADeliveryDefect, listed in the topic map, plus a Category: Fix What's New entry.

Mirror-sync: none — no i18n catalog key is added or re-worded.

No address needs recycling for this: it is framework code in the image, live on the next roll.

🤖 Generated with Claude Code

…ry defect ClassifyDeliveryException reports ShuttingDown for "a lifecycle transition by construction" and Failed for everything else, and the host half of that rule was two exception TYPES -- SiloUnavailableException and OrleansLifecycleCanceledException. The two ways Orleans actually reports a departed pod are neither, so after the six re-resolving retries the sender was told the failure was permanent, and every consumer carrying its own recovery machinery (SynchronizationStream's resubscribe latch, MeshNodeStreamCache's transient-owner rule) tore itself down for a target that was answering again on the surviving pod. The two production shapes, verbatim: Exception while sending message: ...ConnectionFailedException: Unable to connect to S10.244.4.87:11111:146498551, will retry after 585.8766ms The target silo is no longer active: target was S10.244.4.183:11111:146524552, but this silo is S10.244.4.183:11111:146534005 Both clear the classifier's bar because a SiloAddress is generation-stamped: each is a statement about ONE silo INCARNATION, and an incarnation whose socket refuses connections or which has been superseded by a named successor never answers at that address again. The only cure is the re-resolve the retry already performs -- IsTransientFailure's OrleansMessageRejectionException type test has matched both shapes since #2314, so nothing new spins; the defect was a classifier that could not read its input, the same shape as #1742 / #2357 / #3139 / #2451. A bare TimeoutException deliberately still stays terminal: the silo ACCEPTED the connection and did not answer, i.e. plausibly wedged rather than restarting, and demoting that is a resubscribe storm. IsShutdownShaped now walks the exception GRAPH rather than the InnerException line, for the reason IsScopeTeardown already gives -- which fault sits at index 0 of PostFailure's two-transport AggregateException is a race. That narrowness applied to the two pre-existing type tests too. The signal is Orleans' message text, which is weaker than a type and known to be: there is no typed rejection reason, and OrleansMessageRejectionException carries every refusal kind, so accepting the type alone would report genuine defects as transient. Both phrases are pinned against the SHIPPED Orleans assemblies' own string literals -- "Unable to connect to" is a literal of Orleans.Core, "The target silo is no longer active" of Orleans.Runtime -- so an Orleans re-wording reds the pin instead of going silently inert. #2299 and #2307 are ONE root seen through two logs: #2299 is fingerprinted on our own verdict line, #2307 on Orleans' per-attempt addressing log, which our own bounded retry multiplies. #2307's stated defect (a caller resending on a fixed interval without re-resolving) does not exist in src/ -- both retry sites are Defer-cold, bounded and exponentially backed off -- so it is closed as consolidated into #2299. The whole account, including the shapes still open, is in the new Architecture page. Verified: Release -warnaserror on MeshWeaver.Hosting.Orleans, MeshWeaver.Documentation and both touched test projects, 0 Warning(s) 0 Error(s) each. MeshWeaver.Hosting.Orleans.Test 277/277, MeshWeaver.Documentation.Test 601/601 (which hosts the async/static/bridge guards and the topic-map guard). Mutation-measured in three directions, because over-accepting here masks defects: predicate not wired in (the pre-fix answer) -> 5 red: the four production shapes plus the aggregate case phrase dropped, exception type alone -> 2 red: an ordinary Orleans fault and a rejection carrying no departed-silo phrase OrleansException type guard dropped -> 1 red: an application connect failure using the same words Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Copilot review overview

🟡 Changes recommended

Narrow the exception guard to OrleansMessageRejectionException and add the corresponding negative test.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

Use concrete rejection type instead of broad OrleansException guard · New

What changed in this PR

This PR classifies departed-silo delivery failures as transient lifecycle events, preserving subscriptions during rolling deployments.

Changes:

Detects departed-silo Orleans message shapes.
Traverses aggregate exception graphs.
Adds regression tests and architecture/What's New documentation.
Critical issue: the exception guard should target OrleansMessageRejectionException rather than the broader OrleansException, with a corresponding negative test.

File	Summary
`test/MeshWeaver.Hosting.Orleans.Test/DepartedSiloClassificationTest.cs`	Adds classification and safeguard tests.
`src/MeshWeaver.Hosting.Orleans/RoutingGrain.cs`	Adds departed-silo classification logic.
`src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-19-a-roll-no-longer-tears-down-mirrors-on-the-pod-it-replaced.md`	Documents the user-facing fix.
`src/MeshWeaver.Documentation/Data/Architecture/ADepartedSiloIsNotADeliveryDefect.md`	Documents the failure model and rationale.
`src/MeshWeaver.Documentation/Data/Architecture.md`	Adds the architecture topic-map entry.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

rbuergi · 2026-09-19T19:19:13Z

+        e is global::Orleans.Runtime.OrleansException
+        && (e.Message.Contains(SiloEndpointUnreachableMarker, StringComparison.OrdinalIgnoreCase)
+            || e.Message.Contains(SupersededSiloGenerationMarker, StringComparison.OrdinalIgnoreCase));


Agreed, and taking it further than narrowing — because narrowing alone would lose coverage in the one direction this file has an incident about.

Why the base type was there, and why it is still not the answer. IsDirectoryUnstable beside this deliberately matches a BARE OrleansException, and its doc comment says why: CallbackData.HandleRejectionResponse resolves rejection?.Exception ?? new OrleansMessageRejectionException(...) — the carried exception wins — so for a whole class of conditions the caller never receives the rejection type at all. That is #1742/#2357. Narrowing my phrase to OrleansMessageRejectionException and stopping there would re-open exactly that hole for the connect shape, whose carried exception is a ConnectionFailedException.

So the fix is two arms, and neither is the base class. I probed the shipped Orleans 10.3.1: Orleans.Runtime.Messaging.ConnectionFailedException is public and referencable, so it is now accepted by TYPE, with no phrase at all — it is thrown only by ConnectionManager.GetConnectionAsync(SiloAddress), i.e. only ever about a cluster endpoint, which makes the type a strictly stronger signal than the prose and immune to an Orleans re-wording. The phrase test is then guarded by the concrete OrleansMessageRejectionException exactly as you ask, which is what the superseded-generation shape needs (Orleans rejects that one with no carried exception, so the caller does get the rejection).

Net effect on your case: a bare application-level OrleansException quoting these words now stays terminal, and a different Orleans rejection quoting them is only accepted if it really is a rejection.

Negative tests added, as asked — beyond the existing InvalidOperationException one:

ABareOrleansExceptionCarryingThePhrase_StaysTerminal — literally your case, the production connect text on a bare OrleansException.

ARejectionWithoutADepartedSiloPhrase_StaysTerminal is now built from the real 2-arg OrleansMessageRejectionException(message, inner) constructor rather than an uninitialized object, so it pins a real rejection carrying a real unrelated message.

AConnectionFailure_IsAcceptedByTypeWithoutAnyPhrase pins the other half: the type arm must not depend on wording.

Re-measured in four directions after the change: dropping the concrete type arm reds the connect shapes; dropping the phrase arm reds the superseded-generation shape; widening the phrase guard back to OrleansException reds your new negative; dropping the wiring reds all four production shapes. Thanks — this made the predicate both narrower and better covered.

Pushed as 7b755ad202 — and one correction to what I wrote above, because I stated a mutation result before measuring it.

I said "dropping the concrete type arm reds the connect shapes". It does not. The production connect texts still classify through arm 2, because production's rejection wrapper embeds the inner's text — that is the double coverage, working as designed. Measured, dropping arm 1 reds exactly one fact, the one written for it:

mutation red

predicate not wired in (the pre-fix answer) 6 — the 4 production shapes, the by-type fact, the aggregate fact

arm 2 (the phrase) dropped 5 — the 4 production shapes and the aggregate fact

arm 1 (the ConnectionFailedException type) dropped 1 — AConnectionFailure_IsAcceptedByTypeWithoutAnyPhrase

arm 2's guard widened back to OrleansException 1 — ABareOrleansExceptionCarryingThePhrase_StaysTerminal (your case)

both guards dropped, phrase matched on any exception 3 — adds the application-level connect failure

So arm 1's value is not redundancy for the observed samples: it is the carried exception wins path, where the caller gets a bare ConnectionFailedException and there is no wrapper text to read, plus the fact that a type cannot go inert when Orleans re-words a message.

One more thing worth flagging, since it changed the tests: OrleansMessageRejectionException has no accessible constructor at all (not 0-, 1- or 2-arg — I probed all three against the pinned 10.3.1). The facts now build it through Activator.CreateInstance with NonPublic binding rather than substituting an OrleansException, because a base-class stand-in would have quietly made every one of them a statement about the base — the exact width your comment asked me to stop accepting. The helper asserts the instance carries the message before any classification, so none of them can pass having classified something other than what it names.

github-actions · 2026-09-19T19:18:06Z