Bake work-lease: concurrent pods on a fresh identity redundantly compile the same types #1706

Activity

1. rbuergi commented on Aug 21, 2026

   

   rbuergi

   on Aug 21, 2026

   ContributorAuthor

   More actions

   Closing as superseded. The mechanism this asks for exists, is on by default, and the scenario it describes cannot arise on the deployment shape we actually run. Evidence, in the order it matters:

   1. There IS a cross-process work-lease, and it is stronger than the one proposed here.
   BuildProtocolDriver (src/MeshWeaver.Hosting/BuildProtocolDriver.cs) claims Admin/Build keyed by framework fingerprint — exactly the identity half of the proposed (type × identity) key — and every other process completes on the per-fingerprint GO instead of baking:

   "🚨 ONE PROCESS BAKES; the rest SUBSCRIBE TO THE GO. … without coordination every replica on a new image independently finds the same framework-stale cache and starts the same sweep into the same volume — concurrent cold compiles of the SAME NodeType" — DynamicTypePreWarmer.cs:461-470

   It carries every property the issue asks for: a unique per-RUN holder so a restarted pod cannot inherit a dead claim (:70), a heartbeat for the claim's lifetime (:118-122), steal-on-stale in the arbiter (BuildNodeType.ClaimStaleAfter = 10 min), a bake-Job priority so a dedicated baker outranks serving pods (:76-82), and — the fail-open requirement — a follower that is granted the freed claim re-reads the durable witness and bakes rather than parking (OnGranted, :392-415). Default ON: var buildProtocol = true; (DynamicTypePreWarmerHostedService.cs:278).

   It is a whole-sweep claim rather than a per-type one, which for the stated problem ("each compiles the same pending set") is the stronger guarantee, not the weaker one: the followers compile nothing at all.

   2. On the deployment we run, no pod compiles a pending set at boot at all.
   deploy/aks/values.aks.yaml:229 — PreWarm__DynamicTypes: "false", with the reasoning at :195-228:

   "Adoption made the sweep redundant. Once the four satellite repos published under the live framework identity, a prod boot measured compiled=0 alreadyBaked=84 … What is left uncovered compiles on first access, when someone actually reaches it — measured at ~2.4 s per type on the fleet, paid once, by that type's first visitor."

   So the sweep — the only thing that enumerates a pending set — does not run. Adoption and the coverage report do.

   3. The lazy compile-on-access path that replaced it is deduplicated by construction.
   A NodeType's own hub is an Orleans grain (MessageHubGrain, src/MeshWeaver.Hosting.Orleans/MessageHubGrain.cs:44), and PreferLocalPlacement explicitly does not weaken that: "Single-activation semantics are untouched: this only biases where a NOT-yet-activated grain comes to life; an existing activation anywhere still wins." Two pods therefore cannot lazily compile the same type concurrently within a cluster.

   What genuinely remains, if anyone wants it — and it is a different, smaller thing than this issue:

   • Per-type follower granularity. A follower waits for the whole GO; it cannot adopt type-by-type as the leader finishes them. BuildProtocolDriver.OpenChunk (:146-191) already claims a per-chunk sub-node, so per-type chunks would be the natural extension — the arbiter, heartbeat and steal-on-stale come for free.
   • The batch path never flips Pending→Compiling (NodeTypeBatchBake.cs:698-702), so the durable per-type claim IsLiveCompileClaim reads (DynamicTypePreWarmer.cs:1136) is not written by it.

   Neither is the cost this issue was filed about. Reopen (or file the narrower one) if the per-type granularity is wanted; a second, competing lease mechanism beside Admin/Build would be the wrong shape.

   _{Audited against origin/main @ 3d85424 while sweeping the #1660 programme.}
2. rbuergi

   closed this as not plannedon Aug 21, 2026