fix: the build follower waits on facts it can read, not on a notification that cannot cross processes

src/MeshWeaver.Documentation/Data/Architecture/BuildCoordination.md

@@ -249,20 +249,49 @@ dependency. The subscription itself is a remote-path watch and uses the
 `SubscribeWithReEstablish` fault taxonomy: transient faults re-establish; a poisoned or
 deleted root is terminal and loud, never a silent 1 Hz retry loop.
 
-🚨 **That feed is not running yet.** `PostgreSqlChangeListener` is registered but never started in
+🚨 **That feed is not running.** `PostgreSqlChangeListener` is registered but never started in
 either partitioned-PG overload — `AddPartitionedPostgreSqlPersistence` has the `AddHostedService`
 call commented out, and the Aspire overload the portal actually uses never had one. Within a
 cluster the GO still reaches every silo, because `GetMeshNodeStream` resolves to the ONE per-node
 hub that owns `Admin/Build` there; **across clusters nothing propagates it**, so a follower in the
-bake silo's peer cluster observes only what its own hub read at activation. Two consequences worth
-keeping straight:
-
-- the claim arbiter must not depend on notification — hence it READS the durable row on every pass
-  (see above) rather than waiting to be told;
-- `BuildProtocolDriver.FollowGo` subscribes to `ObserveBuildGo` with no bound, so a cross-cluster
-  follower can wait indefinitely for a GO it will never be shown. Starting the listener (or giving
-  the follower a bounded fall-back to its own sweep) is the outstanding work here; until then, treat
-  a *separate-cluster* bake host as "publishes the GO earlier", not as "spares its peers the bake".
+bake silo's peer cluster observes only what its own hub read at activation.
+
+**Nothing in the protocol may therefore depend on being notified.** The arbiter never did — it
+READS the durable row on every pass (above). The follower used to, and that is #1440.
+
+### The follower has two doors, and it closes them behind it
+
+`BuildProtocolDriver.FollowGo` used to be a single `ObserveBuildGo(fingerprint).Take(1)`. That is a
+projection of *this cluster's mirror*, so a cross-cluster follower was waiting on an event that
+could not be delivered to it — and it had also stopped observing its own claim, so the one event
+that could still reach it in-process went unheard. It had no door at all. It now ends on either of
+two **real** events:
+
+1. **The GO becomes visible** — `ReadBuildGo` reads it off the durable build root (the same witness
+   the arbiter decides on) *and* `ObserveBuildGo` watches this cluster's mirror, whichever answers
+   first. The read is what lets a peer cluster's GO mean anything at all here.
+2. **The arbiter hands us the claim.** Registering as a candidate outlives the grant wait, and the
+   arbiter grants the moment the build falls free — whether the builder finished or died. On a grant
+   the follower **re-reads the durable witness**: a GO already there means the winner finished and
+   this process stands down without re-baking; no GO means the builder went away mid-build and this
+   process bakes. The grant is the level-trigger; the witness is the verdict.
+
+**Standing down is not bookkeeping.** `RequestBuildClaim`'s registration survives until a grant
+consumes it, so a follower that reached its answer some other way is later handed a build it will
+never run — holding the claim at `Planning`, never heartbeating, and never taken over, because its
+process is alive and the takeover rule below defends a live holder by design (a stopped heartbeat on
+a live process means busy, not dead). The *next* image's fingerprint could then never be claimed and never get a GO, holding
+every silo's readiness down. `WithdrawBuildClaim` removes the registration and gives back a claim
+that raced it; both halves are conditional, so they are safe against a concurrent grant.
+
+There is deliberately **no timer and no bound bolted onto the wait**. A bound would end the wait by
+guessing, and a follower that guesses "the build finished" certifies a share it never probed — a
+silent wrong answer, strictly worse than a hang that announces itself. Both doors are level-triggered
+on durable state, exactly like the arbiter (and see #1366 for why the poll clock went).
+
+**Starting the listener is a separate decision.** It would make the GO propagate promptly across
+clusters instead of at the arbiter's next pass, but it changes behaviour for every partitioned-PG
+deployment and wants its own justification and measurement. The follower is correct without it.
 
 The probe semantics of `NodeTypeBakeGateState` are preserved unchanged — fail **closed**
 on a measured regression (the rollout stalls, the old image keeps serving), fail **open**

src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-13-a-follower-that-can-never-be-told.md

@@ -0,0 +1,33 @@
+---
+Name: A rollout no longer waits on a message that cannot arrive
+Category: Fix
+Description: A portal that lets another process do the startup compile now works out for itself that the work finished — or takes it over when the other process stopped — instead of waiting for an announcement it could never receive.
+Icon: ArrowSync
+Order: -20260813
+---
+
+# A rollout no longer waits on a message that cannot arrive
+
+When several processes start on a new version, one of them does the compile and the rest wait for
+it to finish. Waiting was the problem: they waited to be *told*, and the announcement is delivered
+only inside the process group that made it. A compile running in a separate group — the dedicated
+build job, or a portal in another cluster sharing the same database — finished, announced, and the
+waiting processes were never reached. Nothing ended the wait.
+
+The same gap swallowed the other outcome too. A process that gives another one the job stays in the
+queue for it, so when the builder **stopped** — crashed, evicted mid-rollout — the job was handed
+straight to a process that had stopped listening. It held the job without doing it, and because it
+was alive and healthy nothing would ever take it away again. The next version could then never
+claim the build at all, and every portal on it stayed out of rotation waiting for a completion
+signal that no longer had an owner.
+
+Waiting has been replaced by looking. The completion signal is a durable record, so a follower now
+**reads** it — the same record the build's own arbitration already decides on — instead of hoping to
+be notified. And it keeps watching for the job: being handed the build is now a real event that ends
+the wait. On being handed it, the follower re-reads the record — finished means stand down without
+repeating a compile that already happened, not finished means the previous builder went away and
+this one takes over. A follower that reaches its answer any other way now gives the job back rather
+than sitting on it.
+
+No timer and no deadline were added. A deadline would end the wait by guessing, and a process that
+guesses "the compile finished" would report a version ready that it never checked.

src/MeshWeaver.Graph/BuildCoordinationExtensions.cs

@@ -8,6 +8,7 @@
 using MeshWeaver.Mesh.Services;
 using MeshWeaver.Messaging;
 using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Logging;
 
 namespace MeshWeaver.Graph;
 
@@ -274,6 +275,13 @@ public static IObservable<MeshNode> FailBuild(
     /// assembly MVID), subscribes here, and holds its health probe not-ready until the emission.
     /// Old-image silos observe their own fingerprint's earlier GO and stay ready throughout a
     /// rollout.
+    ///
+    /// <para>🚨 This is a subscription to THIS cluster's MIRROR of the build root, so it only ever
+    /// emits for a GO this cluster has been told about. A builder in another process publishes to
+    /// the same durable row but no change feed crosses the process boundary (the PG <c>LISTEN</c>
+    /// session is not started in the partitioned wiring), so a mirror activated before that write
+    /// never learns. Anything that must not hang on the answer reads
+    /// <see cref="ReadBuildGo"/> — the durable witness — rather than waiting to be told (#1440).</para>
     /// </summary>
     /// <param name="hub">The calling hub.</param>
     /// <param name="frameworkVersion">The fingerprint to wait for.</param>
@@ -287,6 +295,104 @@ public static IObservable<BuildGo> ObserveBuildGo(this IMessageHub hub, string f
             .Select(go => go!)
             .DistinctUntilChanged();
 
+    /// <summary>
+    /// READS the GO for <paramref name="frameworkVersion"/> off the DURABLE build root — the same
+    /// witness the claim arbiter decides on (<c>BuildNodeType.ArbitrateDurably</c>), and for the
+    /// same reason: it is the ONE record a process in another cluster shares with the builder, so
+    /// it answers where <see cref="ObserveBuildGo"/> can only wait (#1440).
+    ///
+    /// <para>Emits the GO record, or <c>null</c>. 🚨 <c>null</c> is deliberately NOT a claim that
+    /// there is no GO — it is "this call has no GO to give you", which covers three cases the
+    /// caller treats identically: the row carries no GO for that fingerprint; there is no durable
+    /// store at all (the mirror IS the whole world there, and <see cref="ObserveBuildGo"/> is
+    /// complete on its own); or the read FAILED, which is logged loudly. Collapsing them is safe
+    /// only because every caller's <c>null</c> branch is the conservative one — it bakes, into
+    /// content-addressed stores, which costs at worst one redundant build and can never corrupt.
+    /// Fail OPEN, the same asymmetry <c>BuildNodeType.ArbitrateDurably</c> applies, and the reason
+    /// no caller may read <c>null</c> as evidence that a build has not happened. Emits exactly once
+    /// and completes.</para>
+    /// </summary>
+    /// <param name="hub">The calling hub.</param>
+    /// <param name="frameworkVersion">The fingerprint whose GO is wanted.</param>
+    /// <param name="logger">Diagnostics — a failed witness read is reported here.</param>
+    /// <returns>Cold observable emitting the GO record or <c>null</c>, then completing.</returns>
+    public static IObservable<BuildGo?> ReadBuildGo(
+        this IMessageHub hub, string frameworkVersion, ILogger? logger = null)
+    {
+        var storage = hub.ServiceProvider.GetService<IStorageAdapter>();
+        if (storage is null)
+            return Observable.Return<BuildGo?>(null);
+
+        var options = hub.JsonSerializerOptions;
+        return storage.Read(BuildNodeType.RootPath, options)
+            .Take(1)
+            .Select(node => node?.ContentAs<BuildState>(options)?.Ready is { } ready
+                && ready.TryGetValue(frameworkVersion, out var go)
+                ? go
+                : null)
+            .Catch((Exception ex) =>
+            {
+                logger?.LogWarning(ex,
+                    "Build GO: could not read the durable witness at {Path} for {Fingerprint} — "
+                    + "answering 'no GO to give you'. That is NOT evidence the build has not "
+                    + "happened; the caller's null branch is the conservative one (it bakes)",
+                    BuildNodeType.RootPath, frameworkVersion);
+                return Observable.Return<BuildGo?>(null);
+            });
+    }
+
+    /// <summary>
+    /// STANDS DOWN as a claim candidate: removes <paramref name="holder"/>'s pending registration
+    /// and, if the arbiter granted it in the meantime, gives the claim straight back.
+    ///
+    /// <para>🚨 Registering is not free to abandon. <see cref="RequestBuildClaim"/>'s entry survives
+    /// until a grant consumes it, and the arbiter grants to whoever is queued the moment the build
+    /// falls free — so a candidate that stopped listening is handed a build nobody will ever run.
+    /// It then holds a claim with <see cref="BuildStatus.Planning"/>, never heartbeats, and — because
+    /// its process is alive and the cluster says so — is NEVER taken over
+    /// (<c>HolderStillHoldsIt</c> defends a live holder by design). The next image's fingerprint can
+    /// then never be claimed and never gets a GO, which holds every silo's readiness down. A
+    /// candidate that reaches its answer some other way must therefore say so here (#1440).</para>
+    ///
+    /// <para>Both halves are conditional and merge-safe: the registration is removed only if it is
+    /// still ours (its own key — RFC 7396 safe against concurrent candidates), the projection is
+    /// cleared only while we are the named holder and the build has not started, and
+    /// <see cref="ReleaseBuildClaim"/> drops the durable lock only if we own it.</para>
+    /// </summary>
+    /// <param name="hub">The calling hub.</param>
+    /// <param name="holder">The candidate standing down.</param>
+    /// <param name="path">Node path; defaults to the build root.</param>
+    /// <returns>Cold observable emitting the node after the write.</returns>
+    public static IObservable<MeshNode> WithdrawBuildClaim(
+        this IMessageHub hub, string holder, string path = BuildNodeType.RootPath)
+        => hub.GetWorkspace().GetMeshNodeStream(path).Update(curr =>
+            {
+                var state = curr?.ContentAs<BuildState>(hub.JsonSerializerOptions);
+                if (curr is null || state is null) return curr!;
+
+                var registered = state.RequestedClaims?.ContainsKey(holder) == true;
+                // Only a claim we were GRANTED but never started on is ours to hand back. A holder
+                // that is Building is mid-bake and must not clear its own claim from here.
+                var grantedNotStarted =
+                    state.ClaimedBy == holder && state.Status is BuildStatus.Planning;
+                if (!registered && !grantedNotStarted) return curr;
+
+                return curr with
+                {
+                    Content = state with
+                    {
+                        RequestedClaims = registered
+                            ? state.RequestedClaims!.Remove(holder)
+                            : state.RequestedClaims,
+                        ClaimedBy = grantedNotStarted ? null : state.ClaimedBy,
+                        ClaimedByIdentity = grantedNotStarted ? null : state.ClaimedByIdentity,
+                        ClaimedAt = grantedNotStarted ? null : state.ClaimedAt,
+                        HeartbeatAt = grantedNotStarted ? null : state.HeartbeatAt,
+                    }
+                };
+            })
+            .SelectMany(node => hub.ReleaseBuildClaim(holder, path).Select(_ => node));
+
     private static MeshNode NewBuildNode(string path, BuildState? initial)
     {
         var split = path.LastIndexOf('/');