Zero-disruption rolls: bake-gated silo generations, activity-pinned grains #1348

Activity

1. rbuergi commented on Aug 21, 2026

   

   rbuergi

   on Aug 21, 2026

   ContributorAuthor

   More actions

   Verdict: SUPERSEDED — closing

   Audited 2026-08-21 against the live memex deployment. The policy in this issue (serve fast,
   never disturb the user) is unchanged and correct. The mechanism it proposes —
   bake-gated silo generations, a two-generation topology, a placement director routing new
   activations by bake state — no longer addresses a real present cost, because there is no boot bake
   left to gate on.

   The gap analysis dated itself — all three items are now false

   This issue's own "Gap analysis (memex, 2026-08-12)" is the right thing to re-measure, so here it is
   re-measured, on the roll that landed at 08:40Z today:

   "The image bake contributes zero (alreadyBaked=0, #1347): the new pod boots cold and the 76 s
   prewarm sweep races the first visitors"

   ShippedPrebuiltBundles: 50 prebuilt assembly(ies) from 19 shipped bundle(s) under
     /data/prebuilt-bundles/s335e04b229c182637869bebae8687c94 are backed by the assembly store —
     0 adopted now, 50 already current and skipped WITHOUT activating their NodeType hubs — in 00:00:02.38
   
   DynamicTypePreWarmer: ADOPT-ONLY boot complete in 00:00:30.27 — adopted=50 uncovered=59 of 109
     dynamic NodeType(s); nothing was compiled at boot (PreWarm:DynamicTypes!=true).
     framework=s335e04b total=109 baked=50 pending=59 — managed +198 MB, working set +211 MB
   

   Against #1814's measured before-number — "every deploy's first pod compiles all 269 types (~10 min,
   +1598 MB)" — that is ~10 min → 30 s, +1598 MB → +211 MB, and 269 compiled → 0 compiled.

   "The roll is a single-Deployment rolling update: the old pod dies grains-and-all — rule 1 has no
   mechanism at all (preStop/grace only softens the kill)."

   Rule 1 has a mechanism now, and it is #1342. Measured on the same roll: the old pod was
   SIGTERM-scheduled at 08:40:52Z and was still 3/3 Running, serving, and executing application code
   at 09:09:33Z. It kept its grains for the same 29 minutes, for free — MessageHubGrain is
   [PreferLocalPlacement], so a circuit's hubs live on the pod serving that circuit.

   "the sweep warms the most user-visible types … last (#1347)"

   The compiling sweep was retired fleet-wide on 2026-08-17 (a31dd19b8); adoption always runs, and
   ordering within a sweep that no longer compiles is moot.

   Rule by rule

   	rule	state
   1	activity-pinned grains, no teardown while in use	delivered — #1342 + [PreferLocalPlacement], bounded by terminationGracePeriodSeconds
   2	until the bake finishes, allocate on OLD silos	premise gone — boot compiles nothing; there is no bake to wait for
   3	if the bake fails, never switch away from OLD	premise gone — same
   4	only a finished bake opens the new generation	premise gone — same
   5	a recycled grain re-hashes onto the new generation	works; the old pod is out of the Service, so new sessions land on the new pod without a placement director

   Rules 2–4 are all one bet: that a deploy's cost is a bake happening at boot on the serving pod.
   That bet was correct when this was filed and is not correct now. The maintainer's own direction in
   #1660 says the same thing from the other end — "we should not dynamically build at all … built on
   CI and just loading into assembly cache" — and that is what shipped.

   What replaced it

   • A rolled-out pod waits for its sessions instead of cutting them off #1342 — the /drain session drain: preStop holds the pod until its last circuit closes, so
     circuits and their grains survive a roll. This is rule 1, and it is what made a
     two-live-deployments topology unnecessary to get it.
   • fix(#1814): the bake published to an address no portal read — restore the portal's canonical surface + guard the equality #1847 — the bake published to a surface identity no portal read, which is why
     alreadyBaked=0. With that fixed the shipped bundles are found and adopted.
   • fix(graph): an adopted prebuilt build is no longer born dirty (#1834) #1848 / fix: boot follows the module's generation pointer, so landed modules load (#1949) #1954 / fix: write the install-records partition policy durably, so its records are visible (#1950) #1957 — adopted builds are no longer born dirty, boot follows the module
     generation pointer, the install-records partition is durably readable.
   • a31dd19b8 (2026-08-17) — the compiling boot sweep retired; boot is adopt-only.
   • NodeType assemblies are built in CI and LOADED at runtime — no runtime Roslyn, no warm-up window (maintainer-directed) #1660 — where the remaining direction lives, and the right home for anything further.

   What survives, and where it already lives

   59 of 109 types on memex are not covered by the shipped bundles and still pay a Roslyn compile on
   first access — rsalzmann/*, UWDeepfield/*, RFPProcessor/*, i.e. DB-resident user and
   partition types that CI cannot see. That is #1660 Workstream 2 verbatim ("Pre-roll bake Job for
   DB-resident types"), already open and already owned. It does not need this issue, and it does not
   need silo generations — it needs the bake to run before the roll, which is what WS2 is.

   I am deliberately not filing a successor for the two-generation topology. Its three stated
   prerequisites — expand/contract Postgres migrations for the whole overlap window, grain state
   round-tripping both ways, and a second full set of assemblies resident — are all costs of that
   design. The bounded-overlap design that shipped pays none of them.

   Reader six months from now: start at #1660 for the build-on-CI direction, #1342 for why your
   sessions survive a roll, and #1814 for the incident that produced the before-numbers above.
2. rbuergi

   closed this as not plannedon Aug 21, 2026