CD heal ledger: the MODULE half is what has been stalling, not the image set (anchor refusal #3293; per-module identity #3306)

main's HEAD 0fdefa3 is missing part of its deployment image set in ACR, so every self-updating install stays on the previous image. The CD reconciler will attempt to publish it; this issue collects those attempts. Close it once CD is green again.

🔁 CD reconcile attempt 1/3 for 0fdefa3 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 0fdefa3. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed 0da87fa (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7651), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

✅ Healed 0fdefa3 (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7652), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

CD failed on main: run 33728545478 for commit 0fdefa3c68706d6e03eb164ae121f2326ef99450 (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

✅ Healed 149dbbc (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7655), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

🔁 CD reconcile attempt 1/3 for e7f1d69 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for e7f1d69. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed e7f1d69 (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7658), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

CD failed on main: run 33730276201 for commit 149dbbc19e42f783d7ad8822473e95d3e1fa2992 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

✅ Healed e7f1d69 (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7657), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

CD failed on main: run 33734369829 for commit e7f1d699e563b54cc57875b176116d75b71398a3 (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

🔁 CD reconcile attempt 1/3 for 79b4894 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 79b4894. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed 79b4894 (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7661), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

✅ Healed 1e4daff (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7662), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

CD failed on main: run 33739395399 for commit 79b489431dc6363e28f4e552c8770f994376aa7b (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 34079277282 for commit e64689e7fb26cc4509a9c69d78d347b38094b9ab (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

✅ Healed 19b0778 (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7976), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

🔁 CD reconcile attempt 1/3 for b75f501 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for b75f501. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed b75f501 (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7977), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

CD failed on main: run 34085298189 for commit 19b077868c43c8cd1663d63bb8bd378fc4a713b2 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

✅ Healed 525a1c4 (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7981), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

🔁 CD reconcile attempt 1/3 for 395a749 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 395a749. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed f498150 (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7988), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

✅ Healed 395a749 (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7989), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

CD failed on main: run 34091987634 for commit f498150477c4a90a893538b5fb16be8f3a962aa1 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

✅ Healed 67bd3d0 (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7991), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

CD failed on main: run 34092928352 for commit 395a749336b90ba77550f4832b629ddab29389b0 (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

🔁 CD reconcile attempt 1/3 for 6f47c29 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 6f47c29. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

Fixed and closing — the ledger now closes itself (#3549, merged `6f47c295f`)

verify-images' heal step used to comment "Close this issue if nothing else is outstanding" and
close nothing, while gate and alert-on-failure both append to whichever ci-failure issue is
open. So this one absorbed every attempt and every heal from the first day: 330 comments,
four days, a dozen unrelated causes — and its title had to be rewritten by hand twice because the
issue outlived every cause it named.

That is not cosmetic. It removes the only two things an alert says:

property	what it should mean	what this issue made it mean
existence	delivery is broken right now	nothing — the issue is always open
age	how long the outage has lasted	how long ago the first ever outage was

The heal now calls gh issue close. Nothing is lost — this thread and all 330 comments stay
exactly where they are, searchable — and the next failure files a fresh issue (both gate and
alert-on-failure create one when none is open), so from here an alert's title names its own cause
and its age is its own outage's age. Closing also hands the reconciler a fresh heal budget, the
same effect the 🛑 Automatic healing STOPPED comment asks a human for.

🚨 The comment stays fatal; the close is a warning. A lost heal comment hides the delivery
record. A failed close hides nothing — the issue stays open, which is the behaviour that shipped
for months, and the next successful heal closes it, so a transient 403 is self-correcting. A fatal
close would instead red verify-images, which delivery-verdict and alert-on-failure both read
as a broken delivery: an alert about the alerting, on a run that delivered.

It is exercised, not asserted. The step now carries id: heal and takes every value through
env:, so .github/scripts/test-cd-steps.py extracts it from the workflow and executes it
against a stubbed gh. Its cases assert on the recorded gh invocations, never on stdout — a step
that prints the word closing and calls no gh issue close is the defect, and the two are
identical in a log. Negative control, run before merging: deleting the gh issue close line makes
the harness exit 1 naming it; restoring it returns exit 0.

⚠️ On the CD state this issue was last tracking. Closing here is deliberate and is exactly what
the fix now does automatically — it does not assert that delivery is currently healthy. If the
module half is still stalling, the next failed CD run files a new ci-failure issue whose
title, age and comment count describe that failure and nothing else. That is the point of the
change: this thread stopped being able to tell you anything about the state it was named for.

Rationale is committed to
Doc/Architecture/ContinuousDeliveryContract
→ "The ledger must CLOSE itself".

Navigation Menu

Repository navigation

CD heal ledger: the MODULE half is what has been stalling, not the image set (anchor refusal #3293; per-module identity #3306) #3176

308 remaining items

Fixed and closing — the ledger now closes itself (#3549, merged `6f47c295f`)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CD heal ledger: the MODULE half is what has been stalling, not the image set (anchor refusal #3293; per-module identity #3306) #3176

Description

Activity

github-actions commented on Sep 3, 2026

github-actions commented on Sep 3, 2026

github-actions commented on Sep 3, 2026

github-actions commented on Sep 3, 2026

github-actions commented on Sep 3, 2026

github-actions commented on Sep 3, 2026

github-actions commented on Sep 3, 2026

github-actions commented on Sep 3, 2026

github-actions commented on Sep 3, 2026

github-actions commented on Sep 3, 2026

github-actions commented on Sep 3, 2026

github-actions commented on Sep 3, 2026

github-actions commented on Sep 3, 2026

github-actions commented on Sep 3, 2026

308 remaining items

github-actions commented on Sep 7, 2026

github-actions commented on Sep 7, 2026

github-actions commented on Sep 7, 2026

github-actions commented on Sep 7, 2026

github-actions commented on Sep 7, 2026

github-actions commented on Sep 7, 2026

github-actions commented on Sep 7, 2026

github-actions commented on Sep 7, 2026

github-actions commented on Sep 7, 2026

github-actions commented on Sep 7, 2026

github-actions commented on Sep 7, 2026

github-actions commented on Sep 7, 2026

github-actions commented on Sep 7, 2026

rbuergi commented on Sep 7, 2026

Fixed and closing — the ledger now closes itself (#3549, merged 6f47c295f)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Fixed and closing — the ledger now closes itself (#3549, merged `6f47c295f`)