NodeType bake single-silo guard: the lease works, but failover is a 10-minute timer where Orleans membership already knows #1355

The guard EXISTS — and it engaged in production today

NodeTypeBakeLease + BakeCoordination (src/MeshWeaver.Hosting/NodeTypeBakeLease.cs) already
implement "ONE POD BAKES; the rest follow the share". It is invoked from
DynamicTypePreWarmer.BakeOrFollow, not from NodeTypeBatchBake.cs or
DynamicTypePreWarmerHostedService.cs — which is why grepping those two for
leader/primary/elected finds nothing.

It is live in every deployed configuration, not just on paper. BakeCoordination is registered
whenever Deployment:Backend=Filesystem, and that is what AKS
(deploy/aks/values.aks.yaml), ACA (deploy/aca/memex-portal/memex-portal.bicep), compose and
compose-ha all set. The Azure branch of Memex.Portal.Distributed/Program.cs — the only path that
does NOT register it — is selected by no deployment.

Proof from the current memex pod (2026-08-12):

Bake lease ACQUIRED for framework 22825f59 by memex-portal-deployment-77c58c4469-g2ngb
    — this pod bakes; others follow

Acquisition itself is sound: an atomic FileMode.CreateNew on the shared /data/assembly-cache
volume, keyed per framework version, held for the bake's lifetime by Observable.Using and released
on completion/error/unsubscribe. Exactly one process wins.

What is actually weak

Three residual weaknesses, in descending order of how much they matter. None has been observed to
bite, and all three portals currently run one replica, so this is not urgent — but "single silo"
is currently best-effort, not a guarantee, and it is worth being explicit about which.

1. Failover is a fixed 10-minute timer, where Orleans already knows the answer.
StaleAfter = 10 min + FollowPollInterval = 60 s: if the baking pod dies, the fleet waits out a
budget before anyone steals the lease. That budget is a stand-in for "is the holder still alive?",
and Orleans cluster membership answers that authoritatively and immediately. The lease already
stamps its holder as Environment.MachineName — the pod name — so the holder IS resolvable against
membership. Replacing the staleness clock with a membership check (or moving the election onto a
fixed-key grain, whose single activation is an election and whose placement survives a silo
death) makes takeover level-triggered on the truth instead of on a clock.

2. The liveness signal is an SMB file timestamp. File.SetLastWriteTimeUtc /
File.GetLastWriteTimeUtc on Azure Files. Metadata caching can make a heartbeat read stale when it
is not, or fresh when the holder is gone. This is the weakest link in the mechanism and it is
invisible when it misfires — the symptom would be two pods baking, which today only shows up as
duplicated work in the logs.

3. Every I/O error fails OPEN. The catch in TryAcquire returns a lease, i.e. "you may bake".
That is a deliberate and defensible trade ("duplicate work is a cost, a fleet that never compiles is
an outage") — but it is exactly where the single-silo property stops being a guarantee. If it must
become one, that branch is the place, and the replacement has to be something that cannot fail
ambiguously.

Why concurrent bakes are harmful here (beyond wasted CPU)

• Concurrent cold compiles of the SAME type are the storm the sequential ordered sweep exists to
  prevent — four of them on memex (2026-07-28 04:05) dropped six plugin roots to the "did not
  settle" overlay and needed a scale-to-zero.
• Each compile loads a collectible ALC, and MessageHubGrain.OnDeactivateAsync carries a documented
  use-after-unload gap for a grain deactivating on a LIVE silo. More concurrent ALC churn is more
  exposure to that class — a mitigation, emphatically not a fix; the fix is the per-ALC
  in-flight tracking named in that comment (CoreCLR heap corruption: an object's MethodTable word is zeroed; the GC faults walking it (upstream — not fixable here) #613).
• There is also a real correctness hazard, not just waste: two bakers both run
  NodeTypeBatchBake.WriteStamp, which reads the freshest node, applies the compile stamp and
  writes at NextVersion. The monotonic write guard makes the loser's stamp bounce (it already logs
  compile-state stamp … was REFUSED), so the bytes land on the share while the record does not
  name them — the type stays pending and is re-baked next boot. Self-healing, but it is precisely
  the "the bake contributed nothing" shape from NodeType bake: the image contributes ZERO baked assemblies — every pod boot recompiles all 280 types #1347.

Suggested scope

Keep the file lease as the fallback for hosts with no cluster (monolith, tests, dev) and put the
election on Orleans where a cluster exists. Note the overlap with #1348's "bake-gated silo
generations" — an election grain should stay self-contained so the two compose rather than collide.