fix: a transient infrastructure fault at init retires the activation instead of latching it FAILED (#4067, #4068)

src/MeshWeaver.Data/DataContext.cs

@@ -539,6 +539,44 @@ private void SettleInitializationGate(Task allInit)
             return;
         }
 
+        // 🚨 A TRANSIENT INFRASTRUCTURE fault is not a property of this activation (#4067, #4068):
+        // the database was unreachable, a name did not resolve. Latching the hub FAILED for it —
+        // the rejection handler below answering every later request with a terminal failure,
+        // the streams errored, nothing ever re-run — turned a two-minute DNS blip into "this node
+        // is broken until the process restarts" (memex-cloud, 2026-09-12: three DataContexts in
+        // two minutes). A hub that demand routing re-creates is RETIRED instead: the backlog
+        // behind the gate is answered "ask again" with the specific cause, the hub disposes
+        // itself, and the next delivery activates a fresh one whose data sources initialise
+        // against the dependency that has come back. Dispose FIRST so the refusal is classified
+        // ShuttingDown (the reporters read IsShuttingDown), FailGate SECOND so the caller's
+        // error names the database rather than the recycle. The root mesh hub, and any hub not
+        // declared WithReactivationOnDemand, keeps the latch — retiring it would not bring it
+        // back — and its log line says so.
+        if (allInit.IsFaulted && InfrastructureFault.IsTransient(allInit.Exception))
+        {
+            if (Hub.Configuration.ReactivatesOnDemand
+                && Hub.Address.Type != AddressExtensions.MeshType)
+            {
+                var reason = ShutdownNack.RetryForTheAuthoritativeAnswer(
+                    Hub.Address,
+                    null,
+                    "its DataContext initialization met a transient infrastructure fault "
+                    + $"({allInit.Exception!.GetBaseException().GetType().Name}: "
+                    + $"{allInit.Exception.GetBaseException().Message}) and this activation is retired");
+                logger.LogWarning(allInit.Exception,
+                    "DataContext initialization for {Address} met a transient infrastructure fault — retiring "
+                    + "this activation instead of latching it FAILED; the address reactivates on the next "
+                    + "delivery and initializes again. {Reason}", Hub.Address, reason);
+                Hub.Dispose();
+                Hub.FailGate(InitializationGateName, reason);
+                return;
+            }
+            logger.LogError(allInit.Exception,
+                "DataContext initialization failed for {Address} on a TRANSIENT infrastructure fault, but this "
+                + "hub is not re-created on demand (no WithReactivationOnDemand), so it stays FAILED until it "
+                + "is recycled or the process restarts.", Hub.Address);
+        }
+
         Exception? failure = null;
         if (!allInit.IsCompleted)
         {

src/MeshWeaver.Documentation/Data/Architecture/HubInitializationFailure.md

@@ -39,6 +39,91 @@ A hub whose initialization throws must:
    must not deactivate the grain, and a `DeliveryFailure` must never beget another (storm). This is
    the same bypass set `MessageService` applies at the gate.
 
+## A TRANSIENT infrastructure fault retires the activation instead — #4067 / #4068
+
+The FAILED state is the right outcome for a fault that belongs to the activation: a NodeType that
+does not compile, a handler that throws, a data source that is misconfigured. No retry would
+change it, and a terminal answer is honest. It is the wrong outcome in kind for a **transient
+infrastructure fault** — the database unreachable, a host name that did not resolve, a
+connection attempt that timed out. Nothing about the activation is broken; the dependency was
+away for a moment. Latching the activation for it turned a two-minute DNS blip into "this address
+is broken until the process restarts": every later request answered with a terminal
+`DeliveryFailure`, the data streams errored, nothing ever re-run.
+
+Measured: memex-cloud, 2026-09-12 06:19–06:21Z — three `DataContext` initializations failed on
+`SocketException: Name or service not known` inside `NpgsqlConnector.ConnectAsync` and stayed
+FAILED (#4068); memex, 2026-08-23 → 2026-09-12 — 19 `sync/*` BuildupActions failed on
+`NpgsqlException: Failed to connect … ---> TimeoutException` across four pods, each leaving its
+hub refusing (#4067).
+
+### The rule
+
+**A hub that demand routing re-creates does not latch a transient fault — it retires.** Both
+seams — `MessageHub.HandleInitialize` (a BuildupAction faulted) and
+`DataContext.SettleInitializationGate` (a data source's initial load faulted) — ask
+`InfrastructureFault.IsTransient(ex)` first, and for a hub declared
+`WithReactivationOnDemand()`:
+
+1. `Dispose()` — FIRST, so every refusal that follows is classified
+   `ErrorType.ShuttingDown` (the reporters read `IsShuttingDown`) and carries this activation's
+   identity;
+2. `FailGate(gate, reason)` — SECOND, so whatever is parked behind the init gate is answered NOW
+   with the SPECIFIC cause in the owner's refusal vocabulary (`Hub X is shutting down — its
+   initialization met a transient infrastructure fault (NpgsqlException: …) and this activation
+   is retired. The address may reactivate; retry to get the authoritative answer.`) rather than
+   the teardown's generic "Hub is shutting down" later;
+3. **no `InitializationError`, no rejection handler, no errored streams** — the marker describes
+   an activation that stays, and this one is going away.
+
+The next delivery to the address activates a fresh hub whose initialization runs again against
+the dependency that has come back. That is the same contract every recycle already carries, and
+every caller already rides it out: the paced re-probe in `GetMeshNodeOutcome`, the
+resubscribe latch in `SynchronizationStream`, the `OwnerDisposing` re-enqueue on the write path.
+Under a sustained outage each demand-driven activation pays one connection timeout and is
+retired — no latch, and no storm either, because the caller's re-probe is paced and bounded and
+ends in `AddressRecyclingException`, which is the honest answer while the database is away.
+
+`WithReactivationOnDemand()` is declared in the ONE funnel every per-node activation passes,
+`NodeTypeRebindWatcher.WithNodeTypeRebind` (Monolith routing and `MessageHubGrain` alike).
+
+### What is transient
+
+`InfrastructureFault.IsTransient` walks the inner-exception chain (so a reflective wrapper or an
+"initialization failed" wrapper does not hide the cause) for a `System.Data.Common.DbException`
+whose own `IsTransient` says so — every ADO.NET provider classifies its connection failures and
+timeouts there; Npgsql sets it for exactly the two shapes measured, with no provider reference
+needed in core — or a bare `SocketException`. A `TimeoutException` on its own is NOT transient:
+the init time-box mints one for a hang, and a hang is a defect. A provider that leaves
+`IsTransient` false has made its own classification, which is honoured.
+
+🚨 **An `AggregateException` is transient only when EVERY branch is.** A `DataContext` initialises
+its data sources under `Task.WhenAll`, so one aggregate can carry a connection timeout from one
+source and a genuine defect from another; reading "any branch transient" as transient would
+retire the activation, discard the defect, and re-run the same failing initialization on every
+reactivation — a latch traded for a loop. A mixed aggregate keeps the latch, whose recorded error
+still carries the transient branch. The full corpus — both incident shapes, the wrappers, the bare
+`TimeoutException`, the non-transient provider fault, all-transient / mixed / nested / empty
+aggregates, a cyclic chain — is `InfrastructureFaultTest`.
+
+### What keeps the latch, and why
+
+A hub **without** `WithReactivationOnDemand()` keeps the FAILED latch on the same fault, and its
+log line now says so explicitly (*"The cause is a TRANSIENT infrastructure fault, but this hub is
+not re-created on demand … so it stays FAILED until it is recycled or the process restarts"*):
+
+- **the root mesh hub** — built once for the process, disposed only by the host; retiring it
+  would not bring it back. A root whose DataContext cannot reach the store at boot is a
+  process-level condition, and the recovery is the process (readiness, restart). The three
+  `mesh/…` ids in #4068 are three boots of one pod, i.e. exactly that recovery happening.
+- **a hub owned by a live object** — a synchronization stream's `sync/*` sub-hub is re-created
+  by the stream's owner's own recovery (a faulted cache entry is evicted on the next read and
+  re-created), not by routing. Retiring it from underneath the stream would replace a terminal
+  fault its owner already handles with a completion its owner does not.
+
+Pinned by `MeshWeaver.Data.Test.TransientInitializationFaultRetiresTheActivationTest` — both
+seams, plus the control without the declaration. Falsified by reverting the two seams: the
+positive cases then fail with `ErrorType.Failed` (the latch) while the controls stay green.
+
 ## Where it lives
 
 `MessageHub.HandleInitialize` wraps the BuildupAction composition in a **liveness bound plus** a
@@ -54,6 +139,10 @@ return Observable
     .Select(_ => { OpenGate(MessageHubConfiguration.InitializeGateName); return request.Processed(); })
     .Catch((Exception ex) =>
     {
+        if (IsShuttingDown || this.IsTerminatedByScopeTeardown(ex))
+            …                                                   // a recognised shutdown, no failure state
+        if (InfrastructureFault.IsTransient(ex) && TryRetireAfterTransientInitializationFault(ex))
+            return Observable.Return(request.Processed());      // retired, not latched — see below
         var reason = ex is TimeoutException
             ? "a BuildupAction did not complete within …s (a hung dependency or stuck compile)"
             : $"a BuildupAction faulted ({ex.GetType().Name}: {ex.Message})";

src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-13-a-node-whose-database-was-briefly-away-recovers-by-itself.md

@@ -0,0 +1,22 @@
+---
+Name: A node whose database was briefly away recovers by itself
+Category: Fix
+Description: A node hub whose start-up could not reach the database — a connection timeout, a name that did not resolve — no longer stays broken until the portal restarts; the next request brings it back and it initialises again.
+Icon: ArrowSync
+Order: -20260913
+---
+
+# A node whose database was briefly away recovers by itself
+
+When a node's hub started up during a short database or DNS outage, its initialisation failed and
+the hub was marked failed for good: every later request to that node was refused with
+"initialization failed", the node rendered as unavailable, and only a recycle or a portal restart
+brought it back — long after the database had returned.
+
+A start-up that fails on a transient infrastructure fault now retires that activation instead of
+marking it failed. Anyone waiting on it is told the address is shutting down and may be retried,
+the hub disposes itself, and the next request creates it afresh, so its initialisation runs again
+against the database that has meanwhile come back. Faults that really belong to the node — a type
+that does not compile, a handler that throws — are still reported as before, and a hub that
+cannot be re-created on demand (the portal's root hub, a stream's own sub-hub) keeps the previous
+behaviour with a log line that now names the transient cause.

src/MeshWeaver.Graph/Configuration/NodeTypeRebindWatcher.cs

@@ -105,6 +105,13 @@ public static MeshNode WithNodeTypeRebind(
         {
             HubConfiguration = config =>
                 baseConfig(config)
+                    // 🚨 A per-node hub is re-created by demand routing from its node whenever it
+                    // is gone — that is what makes a recycle safe, and it is what lets a
+                    // TRANSIENT initialization fault (the database away, a name unresolved —
+                    // #4067/#4068) retire the activation instead of latching it FAILED until the
+                    // process restarts. Declared HERE because this is the single funnel every
+                    // per-node activation passes (Monolith routing AND MessageHubGrain).
+                    .WithReactivationOnDemand()
                     .WithInitialization(instanceHub =>
                     {
                         // Fire-and-forget by design: a watcher that cannot be armed must never

src/MeshWeaver.Messaging.Contract/InfrastructureFault.cs

@@ -0,0 +1,82 @@
+using System.Data.Common;
+using System.Net.Sockets;
+
+namespace MeshWeaver.Messaging;
+
+/// <summary>
+/// Classifies a fault of the INFRASTRUCTURE a hub depends on — the database could not be reached,
+/// a host name did not resolve, a connection attempt timed out — as distinct from a fault of the
+/// hub's own configuration or code.
+///
+/// <para>🚨 <b>Why the distinction is load-bearing (Systemorph/MeshWeaver#4067, #4068).</b> A hub
+/// whose initialization faults enters a FAILED state that answers every later request with a
+/// terminal <c>DeliveryFailure</c> and is never re-run — the right outcome for a NodeType that
+/// does not compile or a handler that throws, which no retry would change. Applied to a
+/// transient infrastructure fault it is wrong in kind: nothing about the activation is broken, the
+/// dependency was merely away for a moment, and latching the activation turned a two-minute DNS
+/// blip into "this address is broken until the process restarts". Measured on memex-cloud,
+/// 2026-09-12: three <c>DataContext</c> initializations failed on
+/// <c>SocketException: Name or service not known</c> inside two minutes and stayed FAILED; on
+/// memex, 19 <c>sync/*</c> BuildupActions failed on <c>NpgsqlException: Failed to connect</c>
+/// across three weeks, each leaving its hub refusing forever.</para>
+///
+/// <para><b>What counts as transient.</b> A <see cref="DbException"/> whose own
+/// <see cref="DbException.IsTransient"/> says so — every ADO.NET provider classifies its
+/// connection failures, timeouts and serialization conflicts there, so this needs no reference to
+/// a provider (Npgsql sets it for exactly the two shapes measured: a connection timeout and a
+/// name-resolution failure) — or a bare <see cref="SocketException"/> anywhere along the
+/// inner-exception chain, so a reflective wrapper or an "initialization failed" wrapper does not
+/// hide the cause. The walk is bounded and reference-deduplicated like <see cref="ExceptionChain"/>'s.</para>
+///
+/// <para><b>What deliberately does not count.</b> A <see cref="TimeoutException"/> on its own: the
+/// initialization time-box mints one for a hang, and a hang is not a transient dependency fault.
+/// An <c>HttpRequestException</c>: it carries a status for most failures and the callers that
+/// meet it have their own retry contracts. A provider that leaves <c>IsTransient</c> false has
+/// made its own classification, which is honoured.</para>
+///
+/// <para>🚨 <b>An <see cref="AggregateException"/> is transient only when EVERY branch is.</b> A
+/// <c>DataContext</c> initialises its data sources under <c>Task.WhenAll</c>, so one aggregate can
+/// carry a connection timeout from one source AND a genuine defect from another. Reading "any
+/// branch transient" as transient would retire the activation, discard the defect, and re-run the
+/// same failing initialization on every reactivation — a latch traded for a loop. So the aggregate
+/// is transient iff all of its (flattened) branches are, and a mixed aggregate keeps the FAILED
+/// latch, whose recorded error still carries the transient branch for the reader.</para>
+/// </summary>
+public static class InfrastructureFault
+{
+    /// <summary>
+    /// True when <paramref name="exception"/> is — or carries, along its inner-exception chain — a
+    /// transient infrastructure fault: a <see cref="DbException"/> with
+    /// <see cref="DbException.IsTransient"/> set, or a <see cref="SocketException"/>. An
+    /// <see cref="AggregateException"/> qualifies only when every one of its branches does.
+    /// </summary>
+    /// <param name="exception">The fault to classify; may be null.</param>
+    /// <returns><c>true</c> for a fault a fresh attempt could reasonably not meet again.</returns>
+    public static bool IsTransient(Exception? exception)
+    {
+        // Bounded like ExceptionChain: a caller-supplied graph can be cyclic or shared.
+        var seen = new HashSet<Exception>(ReferenceEqualityComparer.Instance);
+        return IsTransient(exception, seen);
+    }
+
+    private const int MaxNodes = 256;
+
+    private static bool IsTransient(Exception? exception, HashSet<Exception> seen)
+    {
+        for (var current = exception; current is not null; current = current.InnerException)
+        {
+            if (!seen.Add(current) || seen.Count > MaxNodes)
+                return false;
+            if (current is AggregateException aggregate)
+            {
+                // Every branch, not any: see the type remarks. Flatten first so a nested aggregate
+                // (WhenAll inside WhenAll) contributes its leaves, not itself.
+                var branches = aggregate.Flatten().InnerExceptions;
+                return branches.Count > 0 && branches.All(branch => IsTransient(branch, seen));
+            }
+            if (current is DbException { IsTransient: true } || current is SocketException)
+                return true;
+        }
+        return false;
+    }
+}