Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .github/workflows/main-cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2674,6 +2674,29 @@ jobs:
platform-image-digest: ${{ needs.plugins-bake-image.outputs.platform_digest }}
bake-publish-targets: ${{ vars.BAKE_PUBLISH_TARGETS }}
platform-ref: ${{ needs.gate.outputs.sha }}
# 🚨 PHASE 4 OF MeshWeaver#3461, core's half. Each publication is written into its own
# run-named directory, sealed there, and `_current` — written after the seal — says which one
# applies. The prefix is never mutated in place again for a reader that follows the pointer,
# which is every portal (`ShippedPrebuiltBundles.PublicationDirectoryOf`) and both Azure-direct
# readers. The flat compatibility copy is still written and still sealed, so a reader that
# cannot follow a pointer sees exactly what it sees today.
#
# 🚨 WHY THIS DOES NOT HAVE TO BE CO-ORDINATED WITH MeshWeaver.Plugins, whose `publish-bake`
# is the OTHER producer of this prefix. Since MeshWeaver#4249 the layout belongs to the PREFIX:
# a run that resolves a live `_current` publishes a generation whatever its caller passed,
# warning by name. So the satellite keeps publishing correctly from the merge of this line
# onward, with no change in its repository and in whichever order the two repos merge.
#
# 🚨 And the precondition that makes that true is MEASURED, not assumed: the mechanism lives in
# the PUBLISHER, so every producer must be running one that carries it. Measured 2026-09-14
# across all six node repos — Plugins, Crm, Education, Reinsurance, SocialMedia, Manufacturing
# — every lane is `@main` and every `scripts-ref:` is `main`, so no pinned publisher and no
# pinned reader of this prefix exists in the fleet. This lane is the one exception and it
# resolves its scripts at `platform-ref` above (`gate.outputs.sha`), i.e. the commit this run
# publishes; on a reconcile run that can be older, and a publisher predating #4249 would then
# refresh the flat copy without moving the pointer — which leaves pointer-following readers on
# the NEWER generation, the direction that is safe. Doc/Architecture/SealedPublicationGenerations.
publication-layout: generation
# Plugins is a ROOT: no upstream to wait for. And it does NOT dispatch its own dependents —
# on a platform release the job below wakes every repo once; a second, package-shaped wave
# from here would wake the same satellites twice for the same identity.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -385,12 +385,41 @@ nowhere else. There is no downstream copy left to fall back on, which makes
**The pin half is still open.** A producer past phase 2 *can* write generations and is still writing
flat. Core CD needs no pin move; the satellites move theirs the way they always do.

### Phase 4 — flip, one prefix at a time *(open)*
### Phase 4 — flip, one prefix at a time *(core's half of `plugins` landed; the satellite's is explicitness, not correctness)*

Set `publication-layout: generation` on every producer of one prefix. The multi-producer prefix is
`plugins` and nothing else: core CD's `plugins-bake` and MeshWeaver.Plugins' `publish-bake`. The
single-producer prefixes each flip in their own repository's PR.

#### What is landed, and the precondition that was measured before it was

`main-cd.yml`'s `plugins-bake` passes `publication-layout: generation`. From that merge, core's
publications of `prebuilt-bundles/<identity>/plugins` are written into their own run-named
directory, sealed there, and pointed at.

🚨 **The prefix-ownership mechanism lives in the PUBLISHER, so "either order" is a claim about
which publisher each producer is RUNNING** — a producer resolving a `publish-bake-bundles.sh` that
predates [#4249](https://github.com/Systemorph/MeshWeaver/pull/4249) still refreshes the flat copy
in place and never moves the pointer. That is the one thing that had to be true before flipping and
it is checkable rather than assumed. **Measured 2026-09-14 across all six node repos** — Plugins,
Crm, Education, Reinsurance, SocialMedia, Manufacturing — every `uses:` of a `node-repo-*.yml` lane
is `@main` and every `scripts-ref:` is `main`. **No pinned publisher and no pinned reader of this
prefix exists in the fleet.** The exception is core's own `plugins-bake`, which passes no
`scripts-ref` and so resolves its scripts at `platform-ref` = the commit the run publishes; on a
reconcile run that commit can be older than #4249, and such a run refreshes the flat copy without
moving the pointer — which leaves pointer-following readers on the **newer** generation, the safe
direction, and is the same flat-copy residual phase 5 removes.

What the satellite's own flip buys is therefore **explicitness, not correctness**: it replaces a
discovered layout (announced by a `::warning::` on every publish) with a declared one.

🚨 **Every reader fails safe to today's behaviour.** `ResolvePublicationPointer` degrades an absent,
blank, unreadable, escaping or dangling pointer to the source directory, where the flat copy still
sits, still sealed — so the worst case of a pointer that cannot be followed is the pre-flip serve.
The single reader that may *not* take that fallback is `node-repo-gate.yml`'s `seed`, because
`download-batch` recurses and flattens; it refuses instead, transiently, and a re-run reads the
publication that now applies.

🚨 **"In one change set" was stated here as the precondition, and it is not available.** The two
producers of `plugins` live in **two repositories**, which cannot merge atomically, and one of them
publishes on every core build. So the flip *must* pass through a prefix where one producer writes
Expand Down Expand Up @@ -524,20 +553,21 @@ was ever visible instead of silently shipping a mixed set.
to 1 on that incident. The **residual is a sibling that has not sealed yet when this run's sweep
ends** (21 seconds, measured), and that is not shrinkable by any amount of checking: it is what
phases 2–5 exist for.
- **The next change is the FLIP, and it is a scope call rather than a code one.** Measured over
- **Phase 4's core half is landed** (2026-09-14) — `main-cd.yml`'s `plugins-bake` passes
`publication-layout: generation`, so `prebuilt-bundles/<identity>/plugins` is on the generation
layout and every reader that follows `_current` stops reading a directory mutated in place. The
satellite's own flip is explicitness, not correctness (phase 4 above). **Phase 5 is open.**
- **The measurement the flip waited on.** Measured over
2026-09-12T08:00Z → 09-13T08:00Z, every `plugins` publish job of both lanes: core `plugins-bake`
**25** executed (23 sealed), Plugins `publish-bake` **10** (8 sealed); **62 of 62** target
verifications read *"N/N file(s) hold this run's bytes … 0 byte-identical from another"*; **0**
MIX refusals; 20 distinct identities, **6 written by both lanes** — all 6 at different Plugins
commits, hours apart, with no overlap in the window. So the defect is **latent, not active**, and
the postcondition is carrying it.
Flipping is still not a core-only change — `plugins`' other producer lives in another repository —
and measured 2026-09-13 **no caller anywhere passes `publication-layout`**, core's `main-cd.yml`
`plugins-bake` included, so the prefix's core-side producer is `flat` too. What has changed is
that it no longer has to be **one** change set: the layout now belongs to the prefix (phase 4
above), so the two producers may flip in either order and the interval between the two merges
cannot produce a stale serve. What phase 3 removed was the reason a flip could not be attempted at
all; what this removes is the reason it had to be attempted atomically across two repositories.
The layout belongs to the prefix (phase 4 above), so the two producers may flip in either order
and the interval between the two merges cannot produce a stale serve. What phase 3 removed was the
reason a flip could not be attempted at all; what #4249 removed is the reason it had to be
attempted atomically across two repositories; core's half is now merged.
- 🚨 **Flipping `plugins` does not by itself stop the publish reds.** The flat compatibility copy is
still replaced in place and still races, so an overlap still costs that copy and still fails the
job — the postcondition covering it is unchanged. What the flip buys immediately is that the
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
---
Name: An update is never read while it is being written
Category: Fix
Description: Installations read the platform's prepared plugin content out of a folder that was being overwritten in place while they read it. Each update is now written into its own folder and only pointed at once it is complete, so a read can no longer catch an update half-finished.
Icon: Checkmark
Order: -20260914
---

# An update is never read while it is being written

When the platform prepares plugin content for an installation, it publishes a set of files —
several dozen of them — that every installation running that platform version then reads. Until
now, publishing an update meant **overwriting that set in place**: remove the "this set is
complete" marker, upload the new files over the old ones, put the marker back.

For the minute and a half that took, the folder held some old files and some new ones. Anything
reading it in that window read a mixture.

## What went wrong

Two different pipelines publish this particular set, and neither can see the other. Interleave them
and the folder ends up marked complete while holding files from **two** different preparations —
self-consistent to everything that reads it, and wrong. The first symptom is an installation that
renders nothing, with a mismatch reported deep in a log.

A byte-level check added earlier turns most of those into a loud refusal rather than a silent mix,
but it is a check performed *after* the fact. It cannot stop the overwrite; it can only notice it.

## What changes

**Each publication is now written into its own folder**, checked there, and only then pointed at.
A one-line pointer says which folder currently applies, and it is moved as the very last step —
after everything in the new folder is verified complete.

So a reader never looks inside a folder that is still being filled in. It follows the pointer,
which either names the previous complete set or the new complete set, and never anything in
between. Two pipelines publishing at the same time now write to two different folders, so there is
no longer a mixture to produce.

**Nothing has to be upgraded to benefit.** A copy in the old flat layout is still published beside
the new one, so an installation that predates this change reads exactly what it read before. If the
pointer is ever missing or unreadable, every reader falls back to that copy — the worst case is the
behaviour of the day before yesterday, not a failure.

## What this does not change

**Older folders are kept.** Nothing is removed until it has been superseded, is named by no
pointer, and is more than a month old.

**One copy still races.** The compatibility copy kept for installations that predate the pointer is
still written in place, so an overlap still costs that copy and is still reported. Retiring it is
the next step, once no installation needs it.
Loading