CD refuses to ship a partial image set, and heals main's HEAD on its own

[rbuergi]

Refs #1026

#1031 made a partial release loud. This makes it impossible, and makes a failed CD run
heal itself instead of waiting for an unrelated merge to paper over it.

Two properties CD did not have:

1. All-or-nothing publication. No image in the set carries a consumer-visible tag unless every
   image in the set built.
2. Self-healing. A failed CD run used to be terminal — nothing re-attempted it, so the commit
   simply never got an image and the hole closed only incidentally, when some later PR merged. That
   incidental repair publishes the later commit, so the failed commit's set is never completed;
   the state does not heal, it is superseded. That is precisely how 4f0c35c went unnoticed.

---

1. All-or-nothing: build everything, publish only after everything built

Each leg now pushes its layers under a per-run staging tag — staging-<sha>-<run_id> — and a
new promote job applies the real tags only after all five legs succeed.

The staging tag is unselectable by construction, not by convention:
VersionSelect.PlatformVersionTag is ^\d+\.\d+\.\d+([-+].*)?$, so a staging-… tag can never be
picked by any self-updater. (That regex is not decorative — it exists because an all-digit git-sha
tag once parsed as version 6943991.0.0 and froze every portal on ci.122.)

Why staging-then-promote, and not build-to-archive

I looked at both. The bytes have to reach the registry for a push to happen at all, so pushing them
to a staging tag costs exactly what pushing them to the real tag costs — the only thing worth
withholding is the tag, because the tag is the entire consumer contract. Promotion is then a
manifest-only operation.

The SDK does support ContainerArchiveOutputPath, but publishing to an archive and pushing later
means moving multi-GB tarballs — full base layers included, ×2 architectures — through GitHub
artifact storage between jobs, and docker load cannot load a multi-arch OCI archive on the
runner's classic image store at all (it would need skopeo/crane as a new dependency). Same
guarantee, an order of magnitude more machinery, and a large new I/O bill on every run.

Rejected outright: one giant sequential job that builds all five then pushes. That is
15.4 + 13.5 + 13.1 + 10 + 2.4 ≈ 55 min of critical path; #979 split these jobs precisely to
avoid that.

I used docker buildx imagetools create rather than az acr import: a manifest PUT is the exact
permission every leg already exercises, and the same primitive already mirrors mw-plugin-test to
GHCR in this workflow. importImage/action is not part of AcrPush, and finding that out on
main is not a good trade.

Rollback: ordered so there is nothing to roll back

Tagging five repositories is not one atomic act. Rather than add a compensating delete that can
fail on its own — and that cannot work anyway for a moving pointer, since untagging latest
destroys it rather than reverting it, and needs an AcrDelete this identity may not hold — the
promotion is ordered so a mid-flight failure is unobservable:

Phase	What it writes
A	identity tags (<version>, <sha>) on every repo except memex-portal-ai:<version>
B	moving pointers: main everywhere, latest on mw-plugin-test, the GHCR mirror
C	memex-portal-ai:<version> — one PUT, the last thing the pipeline does

Phase C is the arming write and nothing else is. SelfUpdateHostedService lists tags for
memex-portal-ai only, picks the newest ^\d+\.\d+\.\d+ one, and patches the portal and
migration Deployments to it. So:

• A or B fails → no consumer-visible partial release. No install can select a version that was
  never written. The residue is inert identity/pointer tags on a commit that did not ship; nothing
  resolves them and the next run supersedes them.
• C fails → a single registry PUT either happened or did not. There is no half-armed state.
• Either way → verify-images still fails, alert-on-failure still files, and the reconciler
  re-drives HEAD within 3 h.

Deliberately not added: a cleanup pass that untags staging. It buys nothing — accumulating tags
per .NET repo per run actually go down, 6 → 5 (staging index + its two RID variants + version +
sha, versus today's version/sha indexes plus four RID-suffixed variants) — and it would add a delete
permission and a second failure mode to the one job that must have neither.

Cost

Measured on real CD runs 31324251944 and 31326034137 (job wall clocks):

Leg	run 31326034137	run 31324251944
nodetype-bake	15m21s (pole)	14m29s (pole)
mw-plugin-test	13m27s	12m15s
portal-ai	13m06s	13m16s
migration	9m58s	13m17s
portal-next	2m23s	2m29s
run total	15m30s	14m38s

Promotion itself, measured tag-to-tag within one repository against the real registry (which is all
the job ever does): portal-ai 2.1 s, portal-next 1.9 s — ~15 s of work inside a ~1 min job.

• Full set available: ~15m30s → ~16m30s (+~1 min, ≈ +7%). The CD critical path grows by that
  one job and nothing else.
• memex-portal-ai's version tag specifically: ~13m06s → ~16m30s (+~3.4 min). The portal now
  waits for bake. That is the policy: 4f0c35c shipped a portal increment the NodeType bake gate
  never certified, and ~3 minutes is what not doing that costs.
• Registry traffic unchanged (same layers, pushed once). Runner time +1 job ≈ +1 min/run.

---

2. Self-healing: a reconciler, not a retry

A schedule (every 3 h) and a workflow_dispatch now enter the same pipeline through gate, which
resolves the target commit and decides whether to publish.

It is driven by observed state, never by a failure it is papering over. It asks "does main's
current HEAD have a complete image set in ACR?" — the same question verify-images asks, answered
by the same script — and only when the answer is no does it drive a publish. It wraps no failing
operation in a retry and suppresses no failure signal: alert-on-failure is untouched and now also
covers gate and promote.

1. What triggers the heal? A 3-hourly schedule, plus manual workflow_dispatch. 3 h sits
comfortably inside the installs' 6 h self-update poll, so a healed image is picked up on the very
next poll. A healthy tick is one ~40 s job — gate stops the moment ACR says the set is
complete — so this costs ~5 min of runner time a day when nothing is wrong.

🚨 The push gate is untouched. The workflow_run path still requires
event == 'push' && head_branch == 'main', which is what stops a fork's pull_request run
(whose head_branch can also be "main") from publishing untrusted code with this repo's secrets.
The new triggers do not read the event payload at all: they resolve the target as the tip of this
repo's main through the API, and re-check Consolidate test results on it. A fork's code can
never be the tip of Systemorph/MeshWeaver's main, so the safety property is preserved by a
stronger check rather than by a proxy. A manual dispatch therefore cannot publish an untested tree.

2. What does it heal to? HEAD, and your instinct is right for a stronger reason than low
value: re-publishing a superseded commit is unsound. The version tag comes from the building
run's number, so republishing older code would mint a higher -ci.<n> for it — and every install
would roll "forward" onto older code, breaking the monotonic-build-number invariant VersionSelect
documents in red ("A non-monotonic build number would make a newer build sort lower and break 'pick
the newest'"). HEAD is also the declared desired state; reconcilers converge on desired state, not
on history. The heal comment on the issue says this explicitly, so nobody re-derives it under
pressure.

A HEAD that cannot ship is not silent either — but it must not cry wolf. The reconcile path
asks the registry first (a complete set means nothing is wrong whatever the check says), and
when the set is incomplete it reads the required check's status as well as its conclusion. Those
two are not interchangeable: conclusion is null both while Build-and-Test is running and when it
never ran at all, so a conclusion-only reading would have posted "main cannot be published" for a
perfectly healthy commit on any tick landing in the ~22 min after a merge — roughly once a day,
until nobody read that issue any more. Still running, or merged minutes ago with no check yet, means
wait, silently, writing nothing. Only a check that has settled on non-success, or one still
absent two hours on (AGENTS.md's trap #1), is the unhealable state — and that is said once per
commit on the ci-failure issue, still refusing to publish. That state clears itself the moment CI goes green on HEAD — and note this also repairs
AGENTS.md's trap #2: the reconciler reads the check on the commit, not the event that produced
it, so gh workflow run "MeshWeaver Build and Test" --ref main now actually leads to an image,
which under the old push-only gate it never could.

3. Does it terminate? Each tick is one attempt and does not re-trigger itself, so a tick
terminates by construction. Persistent failure is bounded at 3 attempts per commit, with the
ci-failure issue as the ledger (the surface a human already watches — no new state store):

• The slot is consumed when an attempt starts, not when it fails, so a run that dies without
  reporting cannot buy infinite retries.
• A new HEAD resets the budget naturally (the marker carries the SHA).
• On exhaustion it stops attempting, says so once (🛑 Automatic healing STOPPED …) and labels
  the issue cd-unhealed. Later ticks then cost ~40 s and print nothing. Loud once, then quiet
  — the opposite of a repeating alarm nobody reads. Closing the issue is a deliberate human act and
  legitimately grants a fresh budget.
• It also refuses to race: if a CD run for the same commit is already in flight, the tick skips
  without consuming a slot.
• A heal that works comments ✅ Healed <sha> on the same issue, so the ledger cannot only grow.

4. Does it interact safely with all-or-nothing? It is literally the same job graph — same
staging tags, same promote gate, same verify-images. A reconcile run cannot publish a partial
set any more than a push run can.

---

Also in here

• portal-next-image now gates on gate like every other leg. It gated on
  workflow_run.conclusion == 'success' — the umbrella conclusion the DELIVERY GATE block exists to
  reject, because one starved job that never ran a step poisons it. That leg alone would then be
  skipped while the other four shipped: silently, the exact partial-release shape of CD failed on main: run 31322066110 #1026. Under
  all-or-nothing it is also the one way a skipped-but-not-failed leg could block the whole set, so
  the two gates must agree.
• One definition of the short SHA (gate's output). git rev-parse --short uses
  core.abbrev=auto and can hand out 8 characters for an ambiguous prefix; a promote that tagged 7
  while verify looked for 8 would be a silent, unreproducible miss.
• .github/scripts/check-image-set.sh — the assertion verify-images makes and the question
  the reconciler asks, in one file, for the same reason shard-assign.sh exists: if the two ever
  disagreed, the reconciler would heal forever or never heal at all.
• The CD alert named an empty commit on a reconcile failure.
  github.event.workflow_run.head_sha is empty on a scheduled run, so the ci-failure issue would
  have read for commit ` ` for exactly the failures the reconciler exists to surface. It now
  takes the commit from gate.
• The GHCR mirror sits in phase B on purpose, so a GHCR outage blocks the release rather than
  silently leaving the plugins repo's latest stale. It is a real coupling and the lesser evil:
  check-image-set.sh only observes ACR, so a GHCR miss placed after the arming write would never
  be reconciled and nothing would notice. Here it fails CD, alerts, and the reconciler re-drives it.
• Reported, not silently fixed: portal-next hand-writes 3.0.0-ci.<n> while every .NET leg
  computes $(Version) = 3.0.0-rc1.ci.<n>, so its version tag has never matched its siblings'.
  Nothing selects it (SelfUpdateOptions only reads memex-portal-ai; deployments pin portal-next
  explicitly) and changing a published tag shape would break whatever is pinned today. It is now
  documented at the line that produces it.
• AGENTS.md — "main-cd.yml has no workflow_dispatch of its own, so CD cannot be kicked
  directly" and "the only thing that produces a tag is a real push to main" are exactly the
  properties this PR removes.
• deploy/CHANGE-PROPAGATION.md — the page a reader consults to learn how a merge reaches a pod
  described neither the required-check gate CD has used since perf(ci): split CD bake job, mirror plugin image to GHCR, add per-project test parallelism #979, nor all-or-nothing, nor the
  reconciler.

No band-aids

No retry around a publish. No dropped architecture. No continue-on-error anywhere (asserted
mechanically). No timeout raised or hidden. A genuine build failure still fails CD loudly and
completely — that is the entire point, and the reconciler's job is to re-attempt HEAD later, not
to make a failure look like a success.

---

Verification

The job graph, parsed with a YAML loader (yaml.safe_load + assertions, not by eye):

gate ─┬─ portal-image ─────┐
      ├─ migration-image ──┤
      ├─ bake-image ───────┼─→ promote ─→ verify-images ─→ (alert-on-failure: failure())
      ├─ plugin-test-image ┤
      └─ portal-next-image ┘

Asserted mechanically: every job transitively depends on gate; alert-on-failure's needs covers
every job including the two new ones; promote needs all five image jobs and carries no
always()/!cancelled(); nothing outside promote pushes a non-staging tag; no
continue-on-error; the push/branch gate is still present on the workflow_run path. All 36 run:
blocks pass bash -n.

The gate blocks — proved end-to-end on real GitHub Actions, not by reading the docs. A throwaway
branch carried a workflow mirroring these exact needs/if shapes with trivial jobs
(run 31328585220, branch since
deleted):

Graph	Setup	Result
A	bake leg fails (the #1026 shape)	a-bake: failure → a-promote: skipped (its body was exit 1 "PROMOTED A PARTIAL SET"; it never ran) · a-verify: success (ran under always()) · a-alert: success (fired)
B	all legs pass	b-promote: success · b-verify: success
C	healthy reconcile tick (publish=false)	every job skipped — no build, no promote, no verify

The reconciler's decision table, driven locally against the Decide step extracted verbatim
from the workflow with a stubbed gh:

push, required check green                      → publish=true   ✅ publishing the full image set
push, required check red                        → publish=false  ⛔ nothing will be built
reconcile, red check but set COMPLETE           → publish=false  ✅ nothing to reconcile
reconcile, CI still running on a fresh merge    → publish=false  ⏳ waiting (SILENT — no write)
reconcile, merged 3 min ago, no check yet       → publish=false  ⏳ waiting (SILENT — no write)
reconcile, no check 2h+ later (blocked)         → publish=false  🚧 + ONE cd-blocked comment
reconcile, check FAILED, 2h+ later (blocked)    → publish=false  🚧 + ONE cd-blocked comment
reconcile, blocked, already reported once       → publish=false  🚧 (silent — no second comment)
reconcile, set already COMPLETE (healthy tick)  → publish=false  ✅ nothing to reconcile
reconcile, incomplete, a CD run in flight       → publish=false  ⏳ skipped (no slot consumed)
reconcile, incomplete, attempt 1 of 3           → publish=true   🔁 + attempt comment
reconcile, incomplete, attempt 3 of 3           → publish=true   🔁 + attempt comment
reconcile, incomplete, budget EXHAUSTED         → publish=false  🛑 + cd-unhealed label + one comment
reconcile, exhausted + already escalated        → publish=false  🛑 (silent — no second comment)
reconcile, incomplete, NO open issue yet        → publish=true   🔁 creates the issue, then attempt 1

Against the real registry (az acr manifest show), reproducing #1026 from ACR state alone:

$ .github/scripts/check-image-set.sh 4f0c35c
memex-portal-ai:4f0c35c (linux/amd64 + linux/arm64)
memex-migration:4f0c35c (linux/amd64 + linux/arm64)
::error::memex-bake:4f0c35c is MISSING from ACR — main 4f0c35c has an INCOMPLETE image set
mw-plugin-test:4f0c35c (linux/amd64 + linux/arm64)
memex-portal-next:4f0c35c (linux/amd64)                                            EXIT=1

$ .github/scripts/check-image-set.sh 86578a4
All five images exist in ACR for 86578a4.                                          EXIT=0

So the reconciler would have fired for 4f0c35c and stayed silent for 86578a4.

Promotion mechanics, on the real registry (probe tags created and removed): imagetools create
tag-to-tag within a repository took 2.1 s (portal-ai) and 1.9 s (portal-next); the
multi-arch source kept its application/vnd.oci.image.index.v1+json with linux/amd64 + linux/arm64, so verify-images' architecture assertion passes on a promoted tag; the single-arch
portal-next is wrapped in a one-entry manifest list, a media-type change every OCI runtime resolves
identically and nothing asserts or pins.

What I could not verify locally: the promote job running under CD's own OIDC identity against
ACR, and a real scheduled tick. The first is a manifest PUT — the same operation every leg's push
already performs with that identity; the second is exercised by the schedule after merge.

What will exercise this change: merging this PR is itself a push to main, so the very next CD
run publishes through the new staging → promote path and verify-images asserts the result. The
reconciler's healthy path runs within 3 h of the merge and should be a single ~40 s gate job
reporting "already has a complete image set". Verify the image, never the tick:

az acr repository show-tags -n meshweaver --repository memex-portal-ai --orderby time_desc --top 5 -o tsv
.github/scripts/check-image-set.sh <short-sha>

No What's New entry

Delivery infrastructure with no user-visible behaviour: nothing in the portal looks or behaves
differently, and the entry categories are Feature / Fix for things a user can notice. The
audience for this change is whoever is on the hook when CD goes red, and AGENTS.md is where they
look — which is why the correction went there instead.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Updates the main-branch CD workflow to prevent partial image publication and to add a reconciler path (scheduled/manual) that can republish main’s HEAD when ACR is missing part of the expected image set.

Changes:

• Introduces a staging-tag → promote flow so consumer-visible tags are only applied after all image legs succeed.
• Adds schedule and workflow_dispatch triggers plus a gate job that resolves the target SHA and decides whether to publish (push vs reconcile).
• Extracts the “complete image set” assertion into .github/scripts/check-image-set.sh and reuses it for both reconcile decision-making and post-publish verification.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
AGENTS.md	Updates operational guidance to reflect staging/promote and reconciler behavior (“verify the image, never the tick”).
.github/workflows/main-cd.yml	Implements all-or-nothing publishing, adds reconcile triggers, centralizes target selection in gate, adds promote job, and reuses a shared image-set assertion.
.github/scripts/check-image-set.sh	New shared script to assert the full expected image set exists in ACR for a given short SHA.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Test Results (shard 0)

685 tests   685 ✅  13m 49s ⏱️
 10 suites    0 💤
 10 files      0 ❌

Results for commit 534f17f.

[github-actions]

Test Results (shard 1)

1 199 tests   1 199 ✅  6m 18s ⏱️
   11 suites      0 💤
   11 files        0 ❌

Results for commit 534f17f.

[github-actions]

Test Results (shard 5)

1 725 tests   1 533 ✅  6m 7s ⏱️
   11 suites    192 💤
   11 files        0 ❌

Results for commit 534f17f.

[github-actions]

Test Results (shard 4)

1 284 tests   1 279 ✅  7m 1s ⏱️
   11 suites      5 💤
   11 files        0 ❌

Results for commit 534f17f.

[github-actions]

Test Results (shard 3)

   11 files     11 suites   6m 15s ⏱️
1 976 tests 1 868 ✅ 108 💤 0 ❌
2 116 runs  2 008 ✅ 108 💤 0 ❌

Results for commit 534f17f.

[github-actions]

Test Results (shard 2)

1 912 tests   1 910 ✅  6m 43s ⏱️
   11 suites      2 💤
   11 files        0 ❌

Results for commit 534f17f.

[github-actions]

Test Results

   65 files     65 suites   46m 16s ⏱️
8 781 tests 8 474 ✅ 307 💤 0 ❌
8 921 runs  8 614 ✅ 307 💤 0 ❌

Results for commit 534f17f.