fix: a transient infrastructure fault at init retires the activation instead of latching it FAILED (#4067, #4068)

[rbuergi]

Fixes #4067. Fixes #4068.

The defect

Both incidents are one mechanism: a hub whose initialization met a transient infrastructure fault entered the FAILED state — InitializationError set, a front-of-chain rule answering every later request with a terminal DeliveryFailure{Failed}, streams errored, nothing ever re-run — until a recycle or a process restart. That latch is the right outcome for a fault of the activation (a NodeType that does not compile, a handler that throws). It is wrong in kind for a fault of the dependency: nothing about the activation is broken, the database was away for a moment, and the latch turned a two-minute DNS blip into "this address is broken until the process restarts".

• Transient DNS failure at PostgreSQL connect time permanently latches mesh DataContext hubs in FAILED state #4068: DataContext.SettleInitializationGate — three DataContexts FAILED on SocketException: Name or service not known in two minutes (memex-cloud, 2026-09-12). The three mesh/… ids are three boots of one pod: the ROOT's recovery is the process, and that is what happened.
• PostgreSQL database at 10.42.18.4:5432 is unreachable — all MessageHub sync initializations fail with connection timeout #4067: MessageHub.HandleInitialize — 19 sync/* BuildupActions FAILED on NpgsqlException: Failed to connect … ---> TimeoutException across three weeks (memex).

Neither is a ReplaySubject latch nor a failed Lazy: it is the FAILED state machine applied without classifying the fault, and no path back.

The fix

1. InfrastructureFault.IsTransient (Messaging.Contract) — DbException { IsTransient: true } (every ADO.NET provider classifies its connection failures there; Npgsql for exactly the two shapes measured, no provider reference in core) or a SocketException, anywhere in the chain (ExceptionChain, so an AggregateException or reflective wrapper does not hide it). A bare TimeoutException is deliberately NOT transient — the init time-box mints one for a hang, and a hang is a defect.
2. WithReactivationOnDemand() on MessageHubConfiguration — "demand routing re-creates this hub after it goes away" — declared in the one funnel every per-node activation passes (NodeTypeRebindWatcher.WithNodeTypeRebind, Monolith routing and MessageHubGrain alike).
3. Both init seams: on a transient fault, a hub declared so is retired instead of latched — Dispose() first (so every refusal is ShuttingDown with this activation's identity), FailGate(reason) second (the parked backlog is answered now with the specific cause: "…its initialization met a transient infrastructure fault (NpgsqlException: Failed to connect …) and this activation is retired. The address may reactivate; retry…"), no InitializationError. The next delivery activates a fresh hub whose initialization runs again against the dependency that has come back — the recycle contract every caller already rides out (paced re-probe, resubscribe latch, OwnerDisposing re-enqueue). Under a sustained outage each demand-driven activation pays one connection timeout and retires; the caller's re-probe is paced and bounded, so no storm and no latch.
4. A hub without the declaration — the root mesh hub (built once for the process; retiring it brings nothing back), a stream's sync/* sub-hub (re-created by its owner's own recovery, not by routing) — keeps the latch, and its log line now says so: "The cause is a TRANSIENT infrastructure fault, but this hub is not re-created on demand … so it stays FAILED until it is recycled or the process restarts."

Test and falsification

TransientInitializationFaultRetiresTheActivationTest (Data.Test, 4 cases, 0.4 s): the first activation's init throws a transient DbException and the second succeeds. Positive cases (both seams): the requester is refused ShuttingDown in the owner's vocabulary, the activation reaches Dead with no InitializationError, the very next delivery is served by a fresh hub, and the init ran exactly twice. Controls (both seams, no declaration): ErrorType.Failed, "initialization failed", the hub stays Started with the marker set, the init ran once.

Reverting the two seams (declaration and classifier kept, so it compiles): the two positive cases fail (found Failed), the two controls pass — the marker, not the fault, selects the retirement.

Also green: MeshWeaver.Data.Test 512/512, MeshWeaver.Messaging.Hub.Test 409/409, the activation/rebind slice of MeshWeaver.Graph.Test (37/37), DocumentationLinkIntegrityTest.

Residue, named

• The root mesh hub keeps the latch by design; its recovery is the process. Nothing in core surfaces a root InitializationError on /health — the pod's restart in Transient DNS failure at PostgreSQL connect time permanently latches mesh DataContext hubs in FAILED state #4068 came from elsewhere (the three boots). A /health entry for the root's DataContext state belongs in the host (Plugins).
• No Plugins change is needed for classification: Npgsql sets IsTransient on NpgsqlException for connection failures, timeouts and name resolution.

Doc: Doc/Architecture/HubInitializationFailure (new section). What's New: Fix.

Pairs-with: none — nothing leaves src/; the additions are a new static class, a new configuration method and a marker record.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BRLdhAiX5DjdZC2WUfPg9E

---

2026-09-13 — the queue red was my TEST, not a #4165/#4151 interaction (fixed in e248b4d63)

• Not an interaction. The failing row is BuildupAction, whose host has no AddData and therefore no SynchronizationStream: its output block in queue build 34750238233 carries zero SYNC_STREAM lines where the passing DataContext row carries five, so fix(layout): a faulted stream re-delivers its fault to a late GetStream subscriber (Plugins#1715) #4151's FaultStore/GetStream change cannot reach it. The steward's solo re-queue then failed on the same row alone against main, which confirms it independently.
• BuildupAction's init faults immediately, so the retire can precede the post. The failing row's own log: TEST START …065 → "retiring this activation" …067 (+2 ms) → FAILED …078. The request activated a fresh hub and was served — exactly what step 3 of the same test asserts as the contract — so Assert.Throws saw a ProbeResponse. Which ordering occurred was decided by how long GetClient() took; a DataContext's init is slower, which is why only that row failed.
• Both orderings are legal and the test now states the whole contract. The first init attempt parks until the test releases it, and the release happens only once the request is provably in flight at that activation; step 3 continues to pin the other ordering. (InFlightAt accepts either queue: fix: a handler that faults past DisposeHostedHubs answers its requester (#4072) #4154's new target pump now: … turn=InitializeHubRequest running 35004ms buffer=1 deferred=0 is what showed that a BuildupAction runs ON the init turn — the request sits in the MAIN queue and never reaches the deferral decision — while a DataContext initialises OFF it and parks in the DEFERRED queue.)

Verified on origin/main merged in (bd4314bc6, with #4151 and #4165): 4/4 green, MeshWeaver.Data.Test 512/512, MeshWeaver.Messaging.Hub.Test 432/432.

[github-actions]

Test Results (shard 0)

262 tests  ±0   262 ✅ ±0   1m 56s ⏱️ +8s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit e248b4d. ± Comparison against base commit bd4314b.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 3)

390 tests  ±0   390 ✅ ±0   55s ⏱️ -4s
  2 suites ±0     0 💤 ±0 
  2 files   ±0     0 ❌ ±0 

Results for commit e248b4d. ± Comparison against base commit bd4314b.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 1)

512 tests  +4   512 ✅ +4   44s ⏱️ +11s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit e248b4d. ± Comparison against base commit bd4314b.

♻️ This comment has been updated with latest results.

[Copilot]

🟡 Changes recommended

Unresolved critical issues remain in mixed-failure handling and transient-classifier coverage.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR enables demand-reactivated hubs to recover from transient infrastructure failures by retiring the affected activation and retrying on the next delivery.

Changes:

• Adds transient infrastructure-fault classification.
• Adds demand-reactivation configuration and retirement behavior.
• Adds regression tests and documentation.

File summaries

File	Summary
test/MeshWeaver.Data.Test/TransientInitializationFaultRetiresTheActivationTest.cs	Tests retirement and failed-latch behavior. Moderate finding (1 vote): assert the DataContext initialization marker directly.
src/MeshWeaver.Messaging.Hub/MessageHubConfiguration.cs	Adds reactivation-on-demand configuration.
src/MeshWeaver.Messaging.Hub/MessageHub.cs	Retires eligible hubs after transient failures. Nit (1 vote): localize the user-facing shutdown reason.
src/MeshWeaver.Messaging.Contract/InfrastructureFault.cs	Classifies transient faults. Critical finding (2 votes): add coverage for excluded, socket, non-transient, aggregate, and mixed exceptions.
src/MeshWeaver.Graph/Configuration/NodeTypeRebindWatcher.cs	Enables reactivation for node activations. Moderate finding (1 vote): test that the composed configuration sets the marker.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-13-a-node-whose-database-was-briefly-away-recovers-by-itself.md	Documents the user-facing fix.
src/MeshWeaver.Documentation/Data/Architecture/HubInitializationFailure.md	Documents initialization recovery. Nit (1 vote): update the stale unconditional-failure example.
src/MeshWeaver.Data/DataContext.cs	Applies recovery to data-source initialization. Critical finding (2 votes): preserve permanent failures in mixed Task.WhenAll results. Moderate findings (1 vote each): avoid duplicate logging and select the matching transient cause. Nit (1 vote): localize the shutdown reason.

Review details

Suppressed comments (8)

src/MeshWeaver.Data/DataContext.cs:578

• For a hub without demand reactivation this logs an Error here and then falls through to the generic fault branch below, which logs the same exception again. That doubles Error/Loki/alert volume for the infrastructure incidents this diagnostic is meant to explain; fold the transient wording into the single final failure log instead of emitting a second event.

            logger.LogError(allInit.Exception,
                "DataContext initialization failed for {Address} on a TRANSIENT infrastructure fault, but this "
                + "hub is not re-created on demand (no WithReactivationOnDemand), so it stays FAILED until it "
                + "is recycled or the process restarts.", Hub.Address);
        }

src/MeshWeaver.Data/DataContext.cs:565

• This what text is also placed directly into DeliveryFailure.Message, so it can be shown to a portal user, but it is a hard-coded English error string. Use the same localized reason mechanism as the other initialization seam and keep only stable protocol data in the shutdown NACK.

                    "its DataContext initialization met a transient infrastructure fault "
                    + $"({allInit.Exception!.GetBaseException().GetType().Name}: "
                    + $"{allInit.Exception.GetBaseException().Message}) and this activation is retired");

src/MeshWeaver.Data/DataContext.cs:565

• Task.WhenAll can contain a transient exception in a non-first branch, but GetBaseException() follows only the first InnerException path. In that case this NACK can describe an unrelated non-transient exception even though the graph classifier retired the activation for a nested DbException/SocketException; extract the matching transient cause from the same graph before constructing the retry reason.

                    "its DataContext initialization met a transient infrastructure fault "
                    + $"({allInit.Exception!.GetBaseException().GetType().Name}: "
                    + $"{allInit.Exception.GetBaseException().Message}) and this activation is retired");

src/MeshWeaver.Documentation/Data/Architecture/HubInitializationFailure.md:64

• The new contract section says transient faults retire the activation, but the later “Where it lives” code block still shows the catch unconditionally calling EnterInitializationFailedState and opening the gate. That stale example omits the classifier and retirement branch, so a maintainer following the documented flow could reintroduce the defect; update the sample or label it as abbreviated.

`InfrastructureFault.IsTransient(ex)` first, and for a hub declared

src/MeshWeaver.Graph/Configuration/NodeTypeRebindWatcher.cs:114

• This is the production wiring that makes the retirement branch reachable for real per-node hubs, but the new transient test applies the marker manually and the existing NodeTypeRebindWatcher coverage only checks that the base delegate ran. A regression removing this call would leave the seam tests green while real node activations still latch FAILED; add an assertion that the composed configuration has ReactivatesOnDemand set.

                    // 🚨 A per-node hub is re-created by demand routing from its node whenever it
                    // is gone — that is what makes a recycle safe, and it is what lets a
                    // TRANSIENT initialization fault (the database away, a name unresolved —
                    // #4067/#4068) retire the activation instead of latching it FAILED until the
                    // process restarts. Declared HERE because this is the single funnel every
                    // per-node activation passes (Monolith routing AND MessageHubGrain).
                    .WithReactivationOnDemand()

src/MeshWeaver.Messaging.Contract/InfrastructureFault.cs:48

• ExceptionChain.Contains is bounded at 256 nodes and uses a LIFO traversal, so an aggregate with more than 256 branches can skip an earlier transient DbException/SocketException and return false. That sends the activation back through the permanent FAILED latch—the defect this classifier is meant to prevent; transient classification needs a complete or targeted traversal that cannot silently drop branches.

        => ExceptionChain.Contains(exception, static e =>
            e is DbException { IsTransient: true } || e is SocketException);

src/MeshWeaver.Messaging.Hub/MessageHub.cs:774

• This what text is carried in DeliveryFailure.Message by the shutdown NACK and can be rendered to a portal user, but it is a hard-coded English error string. Keep the activation/wire data separate and resolve the human-readable reason through the English/German localization catalog at the delivery or presentation boundary.

            "its initialization met a transient infrastructure fault "
            + $"({ex.GetType().Name}: {ex.Message}) and this activation is retired");

test/MeshWeaver.Data.Test/TransientInitializationFaultRetiresTheActivationTest.cs:113

• For the DataContext seam, this assertion reads MessageHub.InitializationError, but the failure marker owned by that seam is DataContext.InitializationError. A regression could set the DataContext marker and still retire the outer hub, so this test would pass; capture the DataContext before disposal and assert its marker is null for Seam.DataContext.

        // 2. That activation is GONE — not parked FAILED with an InitializationError.
        await first.DisposalCompleted.FirstOrDefaultAsync().Timeout(TestTimeouts.Convergence).Await(ct);
        first.RunLevel.Should().Be(MessageHubRunLevel.Dead);
        ((MessageHub)first).InitializationError.Should().BeNull(
            "a retired activation records no FAILED marker — the marker describes one that stays");

• Files reviewed: 8/8 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Test Results (shard 2)

1 356 tests  +18   1 354 ✅ +18   3m 52s ⏱️ -2s
    3 suites ± 0       2 💤 ± 0 
    3 files   ± 0       0 ❌ ± 0 

Results for commit e248b4d. ± Comparison against base commit bd4314b.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 4)

2 036 tests  ±0   2 036 ✅ ±0   5m 1s ⏱️ +6s
    4 suites ±0       0 💤 ±0 
    4 files   ±0       0 ❌ ±0 

Results for commit e248b4d. ± Comparison against base commit bd4314b.

♻️ This comment has been updated with latest results.

[rbuergi]

Addressed in the follow-up commit:

• Mixed aggregate — InfrastructureFault.IsTransient now treats an AggregateException as transient only when EVERY flattened branch is; a mixed one keeps the FAILED latch (whose recorded error still carries the transient branch). The walk is bounded and reference-deduplicated.
• Classifier corpus — InfrastructureFaultTest: both incident shapes (transient provider fault wrapping TimeoutException / SocketException), a bare SocketException, the bare TimeoutException (deliberately NOT transient), a non-transient provider fault, an "initialization failed" wrapper and a reflective wrapper over a transient cause, a genuine fault that mentions the database in its text, all-transient / mixed / nested / empty aggregates, a transient cause under an aggregate branch's wrapper, and a cyclic chain.
• The funnel declares the marker — NodeTypeRebindWatcherTest.NonNullHubConfiguration_IsComposedNotReplaced asserts the composed per-node configuration reads ReactivatesOnDemand.
• Doc — the HandleInitialize excerpt now shows the retire arm; the aggregate rule and the corpus are documented.
• Not changed: the shutdown reason is a wire/log string in the owner's refusal vocabulary (ShutdownNack), matched by the transient classifiers — like every other ShutdownNack banner it is deliberately not localized (it is not a viewer surface).

[github-actions]

Test Results (shard 5)

    6 files  ±0      6 suites  ±0   8m 38s ⏱️ +10s
2 912 tests ±0  2 720 ✅ ±0  192 💤 ±0  0 ❌ ±0 
2 916 runs  ±0  2 724 ✅ ±0  192 💤 ±0  0 ❌ ±0 

Results for commit e248b4d. ± Comparison against base commit bd4314b.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results

   17 files  ± 0     17 suites  ±0   21m 9s ⏱️ +30s
7 468 tests +22  7 274 ✅ +22  194 💤 ±0  0 ❌ ±0 
7 472 runs  +22  7 278 ✅ +22  194 💤 ±0  0 ❌ ±0 

Results for commit e248b4d. ± Comparison against base commit bd4314b.

♻️ This comment has been updated with latest results.

[meshweaver-cloud]

🔀 Merge-queue steward: re-queued alone (bisecting) — bisecting: your group of 3 pull requests failed on an uncatalogued assertion while this PR's own run was green; re-queued ALONE — if the solo build fails too, the PR stays out (group: #4165, #4151).
Group build: https://github.com/Systemorph/MeshWeaver/actions/runs/34750238233

• MeshWeaver.Data.Test.TransientInitializationFaultRetiresTheActivationTest.ATransientFault_RetiresTheActivation_AndTheNextDeliveryIsServedByAFreshOne(which: BuildupAction) — Assert.Throws() Failure: No exception was thrown

enqueued

[meshweaver-cloud]

ℹ️ Merge-queue steward: no action — removed from the queue with reason MANUAL — the steward takes no action for this reason.

[meshweaver-cloud]

🔀 Merge-queue steward: re-queued alone (bisecting) — bisecting: your group of 2 pull requests failed on an uncatalogued assertion while this PR's own run was green; re-queued ALONE — if the solo build fails too, the PR stays out (group: #4172).
Group build: https://github.com/Systemorph/MeshWeaver/actions/runs/34751989267

• MeshWeaver.Graph.Test.DanglingNodeTypeUpdateTest.Upsert_WithTheNamedImportEscapeHatch_StillLands — MeshWeaver.Reactive.Assertions.AssertionException : Expected value to be true because retyping a stranded node to a type that DOES resolve is the sanctioned repair. Error was: Inner UpdateNode faulted: MeshNode OwnerUnreachable at 'TestData

enqueued