fix(build): the build claim is exclusive across clusters, not just within one

src/MeshWeaver.Documentation/Data/Architecture/BuildCoordination.md

@@ -33,7 +33,9 @@ The protocol below removes all three **by construction** rather than by compensa
 
 ```
 Admin/Build                       ← the build ROOT (durable node, nodeType Build)
+Admin/Build/_Claim                ← the root's claim LOCK (durable; only atomic writes)
 Admin/Build/{chunkName}           ← one node per CHUNK (durable, nodeType Build)
+Admin/Build/{chunkName}/_Claim    ← the chunk's claim LOCK
 Admin/Build/{chunkName}/_Activity ← the chunk's execution activity (standard Activity)
 {nodeTypePath}/Release/{version}  ← unchanged: releases minted per compiled NodeType
 ```
@@ -57,31 +59,97 @@ files, no request/response types, no in-memory gates. Writers go through
 
 ## Who becomes the build master
 
-Nobody is elected. **Mastership is a claim written into the root node**, arbitrated the
-same way every concurrent write in this system is arbitrated: inside the `Update` lambda.
-
-```csharp
-stream.Update(node =>
-{
-    var b = node.ContentAs<BuildRoot>(hub.JsonSerializerOptions);
-    // The claim check and the claim write are one serialised step on the owning hub.
-    if (b is null || (b.ClaimedBy is not null && b.FrameworkVersion == myFingerprint))
-        return node;                    // someone else already owns this build — bail
-    return node with { Content = b with
-        { ClaimedBy = myIdentity, FrameworkVersion = myFingerprint, Status = BuildStatus.Planning } };
-});
-```
-
-The owning hub's action block serialises every writer, so the first candidate's lambda
-sees an unclaimed build and takes it; every later candidate's lambda re-reads the claimed
-state and returns the node unchanged. The claimant then observes its own claim on the
-stream before doing any work — a claim you cannot read back is a claim you do not hold.
-In-memory single-flight flags are permitted only as coalescers; **correctness comes from
-node state** (the [ActivityControlPlane](../ActivityControlPlane) rule).
+Nobody is elected. **Mastership is a claim written into the root node.** Candidates register
+under their own holder id in `RequestedClaims` (per-candidate keys — RFC 7396 merge-safe, so
+concurrent registrations compose instead of overwriting each other), and the node's own hub
+arbitrates. The claimant then observes its own claim on the stream before doing any work — a
+claim you cannot read back is a claim you do not hold. In-memory single-flight flags are
+permitted only as coalescers; **correctness comes from node state** (the
+[ActivityControlPlane](../ActivityControlPlane) rule).
 
 The same claim shape applies per chunk, which is what makes parallel builders safe later:
-each would-be builder claims `Admin/Build/{chunkName}`; the owning hub hands each chunk
-to exactly one.
+each would-be builder claims `Admin/Build/{chunkName}`; exactly one gets each chunk.
+
+### 🚨 The grant is taken on the DURABLE ROW — a hub lambda is only exclusive within ONE cluster
+
+The obvious implementation is to decide inside the owning hub's serialised `Update` lambda:
+the action block orders every writer, so the first candidate's lambda sees an unclaimed build
+and takes it, and every later lambda re-reads the claimed state and bails. That is correct —
+**and it is exclusive only within one Orleans cluster**, which is not the topology this
+protocol runs in.
+
+A second cluster over the same database activates its *own* hub for `Admin/Build`, runs its
+*own* arbiter against its *own* mirror, and grants its own candidate. Three things then line
+up to make the collision invisible (#1424):
+
+- both writes are minted at the same next `MeshNode.Version`, and the store's monotonic
+  condition **applies at equal versions** — re-persisting an unchanged node is a legitimate,
+  common shape — so both land, last-write-wins;
+- neither writer is told it lost: the "store refused" signal is `saved.Version >
+  written.Version`, and the versions are *equal*;
+- nothing propagates the other cluster's write back. Orleans membership and its memory
+  streams are per-cluster by construction, and there is no cross-process change feed running
+  (see the GO section below), so a mirror can be arbitrarily far behind and never learn.
+
+On 2026-08-13 that is exactly what happened on `memex-cloud`: the ephemeral bake Job and the
+rolling serving pod each claimed `Admin/Build` and each ran the full 268-type bake.
+
+#### The claim lives on a LOCK, not on the Build node
+
+Making the *grant* exclusive is necessary and not sufficient, and the second half is the less
+obvious one. Every cluster mirrors `Admin/Build` in its own workspace, and that mirror is
+flushed to storage as a **whole node** by the ordinary persistence sampler — a plain,
+unconditional write from a hub that may never have held the claim. `MeshNode.Version` is a
+per-node counter, not a cross-cluster logical clock, so both mirrors climb it independently and
+the later flush simply wins the row. A *losing* cluster therefore overwrites the winner's
+`ClaimedBy` with its own `null`, its arbiter sees a free build on the next pass, and the
+exclusivity a compare-and-set had just established is undone by a write that knows nothing about
+it. (This is observed behaviour, not a hypothesis: the cross-cluster test caught exactly it.)
+
+So the claim moved off the contended row onto a **lock node**, `Admin/Build/_Claim` (and
+`Admin/Build/{chunk}/_Claim`). The lock has no hub, no mirror and no sampler. Exactly two things
+ever write it, and neither is unconditional:
+
+| Operation | Primitive | Exclusivity |
+|---|---|---|
+| grant / takeover | `IStorageAdapter.WriteIfVersion(node, expectedVersion)` | compare-and-set on the version the arbiter read (`0` = "must not exist") |
+| heartbeat | `WriteIfVersion` | no-op unless this holder still owns the lock |
+| release | `IStorageAdapter.DeleteIfExists` | rowcount-gated, first delete wins |
+
+`BuildState.ClaimedBy` on the Build node stays as the **observable projection** — what the GUI
+and `ObserveBuildClaim` read. A flush clobbering it now costs a stale view in one cluster, never
+a second builder, because no decision is taken from it.
+
+#### The three steps, in this order
+
+`BuildNodeType.ArbitrateDurably`:
+
+1. **Read the lock.** Not the mirror — the mirror is per-cluster and cannot see a rival's grant.
+2. **Decide** with the unchanged `Arbitrate`, over the *lock's* holder state and *this cluster's*
+   pending registrations (from its mirror, because a candidate that registered milliseconds ago
+   has not reached storage yet — and a cluster can only ever grant one of its own candidates).
+3. **Commit with the compare-and-set** against the version step 1 read. At most one of N
+   concurrent arbiters is told `true`. Only then is the grant published on the Build node.
+
+Step 3 has to come before the mirror write, not after, because **the mirror commits ~200 ms
+before the persistence sampler reaches storage** and `ObserveBuildClaim` emits off the mirror.
+Adjudicating after the fact would find the loser already baking.
+
+Losing writes nothing and retries nothing: a refused compare-and-set means another cluster's
+grant is already durable, so this cluster does not grant, its candidate runs out its
+`GrantWindow` and follows the GO — the path the protocol already has for "held elsewhere".
+The arbiter is level-triggered, so the next legitimate trigger re-decides against fresh
+durable state. No timer, no election, no backoff loop.
+
+**It fails OPEN.** With no storage provider owning the path (or no persistence at all — a
+monolith, a test, a dev box) the grant is taken on the mirror exactly as before. Being wrong
+in that direction costs one duplicated bake: bounded and non-corrupting, because every
+downstream write is content-addressed (assembly store keyed by content, release versions
+minted from content hashes, GO keyed by fingerprint). Being wrong the other way — refusing to
+grant because exclusivity could not be proven — leaves the fingerprint with no GO at all,
+which holds every silo's readiness probe down and stalls the rollout. Same asymmetry the bake
+gate applies, and the opposite of `AssemblyCacheRetention`, where the wrong answer deletes
+bytes a running pod still needs.
 
 ### 🚨 When may the claim be taken away — cluster MEMBERSHIP decides, not a clock
 
@@ -108,6 +176,14 @@ runs probes, indirect probes and a membership table for precisely this. So the c
 cluster to ask — a monolith, a test, a dev box, the Orleans *client* host — plus a claim written
 before identities were stamped and an identity membership cannot resolve.
 
+🚨 **A holder in ANOTHER cluster is always `Unknown`**, because Orleans membership is per-cluster:
+the bake silo's ServiceId is not in the serving cluster's table and vice versa, and absence is
+`Unknown`, never `Gone`. So a cross-cluster takeover falls back to the `ClaimStaleAfter` clock —
+this is the one place the clock still governs a live fleet, and it is deliberate: the alternative
+is treating "I cannot see you" as "you are dead", which is precisely the eviction of a live builder
+the membership rule exists to prevent. What #1424 adds is that the takeover WRITE is itself a
+compare-and-set, so two clusters that time out on the same dead holder cannot both succeed it.
+
 🚨 **Absence from the membership snapshot is `Unknown`, never `Gone`.** Orleans keeps departed silos
 in the table as `Dead` until the defunct-cleanup window elapses, so a silo that really died IS in the
 snapshot and resolves to `Gone`. One that is missing entirely usually means our own snapshot is not
@@ -165,14 +241,29 @@ probe accordingly:
   written when *their* image baked, and a newer build's state transitions never revoke an
   older GO — the root's `Ready` is per-fingerprint history, not a global boolean.
 
-The broadcast rides the **durable store's change feed** (Postgres `LISTEN/NOTIFY` via
-`PostgreSqlChangeListener`), which is cross-process and therefore cross-ServiceId by
+The broadcast is *designed* to ride the **durable store's change feed** (Postgres `LISTEN/NOTIFY`
+via `PostgreSqlChangeListener`), which is cross-process and therefore cross-ServiceId by
 construction. It deliberately does *not* ride the Orleans stream relay — the bake cluster
 is not in the portal's cluster, and intra-cluster relays must not be a readiness
 dependency. The subscription itself is a remote-path watch and uses the
 `SubscribeWithReEstablish` fault taxonomy: transient faults re-establish; a poisoned or
 deleted root is terminal and loud, never a silent 1 Hz retry loop.
 
+🚨 **That feed is not running yet.** `PostgreSqlChangeListener` is registered but never started in
+either partitioned-PG overload — `AddPartitionedPostgreSqlPersistence` has the `AddHostedService`
+call commented out, and the Aspire overload the portal actually uses never had one. Within a
+cluster the GO still reaches every silo, because `GetMeshNodeStream` resolves to the ONE per-node
+hub that owns `Admin/Build` there; **across clusters nothing propagates it**, so a follower in the
+bake silo's peer cluster observes only what its own hub read at activation. Two consequences worth
+keeping straight:
+
+- the claim arbiter must not depend on notification — hence it READS the durable row on every pass
+  (see above) rather than waiting to be told;
+- `BuildProtocolDriver.FollowGo` subscribes to `ObserveBuildGo` with no bound, so a cross-cluster
+  follower can wait indefinitely for a GO it will never be shown. Starting the listener (or giving
+  the follower a bounded fall-back to its own sweep) is the outstanding work here; until then, treat
+  a *separate-cluster* bake host as "publishes the GO earlier", not as "spares its peers the bake".
+
 The probe semantics of `NodeTypeBakeGateState` are preserved unchanged — fail **closed**
 on a measured regression (the rollout stalls, the old image keeps serving), fail **open**
 when nothing is measuring (a configuration mistake must never black-hole a pod), gate

src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-13-two-servers-no-longer-run-the-same-build-twice.md

@@ -0,0 +1,31 @@
+---
+Name: Two servers no longer run the same build twice
+Category: Fix
+Description: A build claim is now exclusive across every server sharing a database, not just within one cluster — so a rollout compiles the platform's dynamic content once instead of once per server.
+Icon: Sparkle
+Order: -20260813
+---
+
+# Two servers no longer run the same build twice
+
+When a new version of the platform is deployed, every piece of dynamic content in it has to be
+recompiled before any server may serve. One server is supposed to claim that work and the rest wait
+for its result.
+
+The claim held only within a single server cluster. Two servers that share a database but belong to
+different clusters each kept their own view of the build, each concluded that nobody had claimed it,
+and each wrote its claim — and neither was told it had lost the race, because the two writes were
+indistinguishable to the database. On the last deployment that meant the whole content set, 268
+items, was compiled twice: once by the dedicated build server that exists precisely to spare the
+live server that cost, and once by the live server anyway.
+
+Claims are now settled in the database itself, by a write that can only succeed for one claimant.
+The server that loses is told so, skips the build, and waits for the winner's result — the behaviour
+that was always intended when a build is already running elsewhere. The claim also moved onto a
+record of its own, so that a server which never held it can no longer overwrite the holder's claim
+when it saves its own copy of the build state.
+
+Nothing about the build's output changes: it was never at risk, because compiled artefacts are
+stored under a key derived from their content, so two servers producing the same build produced the
+same bytes. What is saved is the duplicated work — and, on the live server during a rollout, the
+memory it took.