CD: main 9072713 has an incomplete image set

main's HEAD 9072713 is missing part of its deployment image set in ACR, so every self-updating install stays on the previous image. The CD reconciler will attempt to publish it; this issue collects those attempts. Close it once CD is green again.

🔁 CD reconcile attempt 1/3 for 9072713 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 9072713. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed 9072713 (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7159), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

🔁 CD reconcile attempt 1/3 for e468d14 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for e468d14. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed e468d14 (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7166), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

🔁 CD reconcile attempt 1/3 for 159a667 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 159a667. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed 159a667 (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7184), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

🔁 CD reconcile attempt 1/3 for 98195f9 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 98195f9. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed 98195f9 (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7189), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

🔁 CD reconcile attempt 1/3 for ec1ef1b (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for ec1ef1b. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

CD failed on main: run 33449855743 for commit ec1ef1b099e0a5e119d89edc15f485d54d20c1d3 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

✅ Healed ec1ef1b (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7201), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

CD failed on main: run 33450649140 for commit ec1ef1b099e0a5e119d89edc15f485d54d20c1d3 (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33453418553 for commit 2758e8c3471ed352f3452fa202c7548aedc6f5fa (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33454649794 for commit f1ed40418c3ba14e0c782f5601d6f8636b18f8c8 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

Confirmed by controlled experiment, and the stall is cleared

The RCA above predicted that the merging IDENTITY is the whole variable. Merging #3002 by hand
(gh pr merge 3002 --merge, as rbuergi) produced merge commit 7965504b at 19:19Z, and:

$ gh run list --branch main --event push --limit 5
19:19  7965504b  in_progress  Hosting Operator
19:19  7965504b  in_progress  MeshWeaver Build and Test
19:19  7965504b  completed    Chart Gate
15:13  8cc1cc6e  completed    Hosting Operator
15:13  8cc1cc6e  completed    Chart Gate

All three push lanes fired — the same three that produced nothing for 19628536, b3e6ae65,
04d25efa and dc6c5a45. Same repository, same branch, same workflow files, same required check;
the only thing that changed between the silent runs and this one is who performed the merge. Note
the gap in that listing is exactly the outage window: 15:13 → 19:19, with four merges inside it and
no runs at all.

So the mechanism is not inferred from documentation, it is measured in both directions:

merged by	push lanes fired
`github-actions[bot]` (armed with `GITHUB_TOKEN`)	none, 4 for 4
`rbuergi` (human)	3 of 3, twice

Consolidate test results is now running on main's HEAD, so CD's readiness gate has the evidence it
has been waiting for since 15:13 and the next tick can publish. The backlog is cleared as a side
effect, not as the fix — the fix is #3003, which stops the arm lane from producing non-triggering
merges in the first place.

⚠️ Two things to keep in mind while that lands:

Every PR merged before fix(ci): the arm lane merges with an App token, so main's push lanes run (#2916) #3003 lands must be merged by a human/PAT identity, or main stalls
again. auto-arm.yml is served from the BASE branch on pull_request_target, so it keeps arming
with GITHUB_TOKEN until the fix is on main — including on fix(ci): the arm lane merges with an App token, so main's push lanes run (#2916) #3003 itself, whose auto-merge I
have disarmed for exactly this reason.
fix(ci): the arm lane merges with an App token, so main's push lanes run (#2916) #3003 needs a grant before it can arm anything: meshweaver-cloud currently holds only
contents: write + metadata: read, and arming requires pull_requests: write. Until that is
granted and accepted on the org installation, the arm job fails red naming it — which is the
intended shape and does not block merging, since Consolidate test results is the only required
context.

And the bake identity mismatch quoted in the RCA is untouched; it will re-appear as soon as commits
start reaching the bake leg again.

Merge pull request #3003 from Systemorph/fix/2916-auto-arm-must-not-m…

CD failed on main: run 33549149030 for commit 7965504b844e0b944bf0fde149ca75293a46d6f7 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

🔁 CD reconcile attempt 1/3 for 30dda84 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 30dda84. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

CD failed on main: run 33553391045 for commit 3aeae8337c0cd7d5e930ea863601cc43f5efc511 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

✅ Healed 30dda84 (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7468), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

CD failed on main: run 33555526073 for commit 30dda849febd95c82d6f896e5dab3baac330c8de (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

🔁 CD reconcile attempt 2/3 for 30dda84 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 30dda84. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed 30dda84 (run) — main's HEAD now has the complete image set (portal 3.0.0-rc9.ci.7471), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

CD failed on main: run 33561120011 for commit 30dda849febd95c82d6f896e5dab3baac330c8de (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33566326021 for commit 30dda849febd95c82d6f896e5dab3baac330c8de (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33570912103 for commit 30dda849febd95c82d6f896e5dab3baac330c8de (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

Delivery is restored — verified at the registry, not by a green tick

$ az acr repository show-tags --name meshweaver --repository memex-portal-ai --orderby time_desc
3.0.0-rc9.ci.7471
main
30dda84-p1d872f2
30dda84
staging-30dda84-33561120011
…

and the three tags that must agree, do:

main                sha256:037d7b9a8dc2708917d8fb1cc2eea8404e5e2ff5f49d9c32a55aa5949b568384
30dda84             sha256:037d7b9a8dc2708917d8fb1cc2eea8404e5e2ff5f49d9c32a55aa5949b568384
3.0.0-rc9.ci.7471   sha256:037d7b9a8dc2708917d8fb1cc2eea8404e5e2ff5f49d9c32a55aa5949b568384

One digest across the version tag, the commit tag and main is the promoted-set signature. Self-updating installs have something newer to take.

Deliberately checked this way rather than by reading a CD run's conclusion: a run can conclude success with every publishing job skipped, and the run that actually promotes is often still in_progress. The registry is the only thing that cannot be wrong about what shipped.

What was actually wrong — and it was never the image jobs

The title says "main 9072713 has an incomplete image set". That framing sent every reader to the image jobs, which were innocent. The image jobs never ran, because CD was waiting for evidence that could not arrive.

auto-arm.yml armed auto-merge with secrets.GITHUB_TOKEN. GitHub performs an auto-merge as the identity that armed it, and a push created with GITHUB_TOKEN deliberately does not trigger workflow runs. So main's HEAD received no push-event run of anything — not MeshWeaver Build and Test, not Chart Gate, not Hosting Operator — therefore no Consolidate test results, therefore main-cd's readiness gate read absent/none and waited, on every tick, for six hours.

Measured in both directions:

merged by	`push` lanes fired
`github-actions[bot]`	none, 4 for 4 (`19628536` `b3e6ae65` `04d25efa` `dc6c5a45`)
a human/PAT identity	3 of 3, repeatedly

Every dashboard was green throughout, because a suppressed trigger is indistinguishable from a slow one — the AGENTS.md skip-trapdoor rule one level down. Absence is the one thing polling cannot age out of.

Fixed in #3003: the arm mints a GitHub App installation token, asserts the credential RED, and has no fallback. ArmedMergeMustTriggerMainsPushLanesGuard fails the build if any workflow merges or arms a PR with GITHUB_TOKEN. The full mechanism is written up in Doc/Architecture/ContinuousDeliveryContract → "The third state the reconciler cannot see — a SUPPRESSED trigger".

Two things that remain open, so closing this does not bury them

⚠️ Arming is still disabled fleet-wide until meshweaver-cloud is granted Pull requests: Read and write (and the org accepts it). The mint fails with "The permissions requested are not granted to this installation", so Arm auto-merge is red on every PR — by design, and it blocks nothing, since Consolidate test results is the only required context. Until then every PR needs a human merge, which is what restored delivery here.
The bake identity mismatch is a separate, still-live defect — af817d9a promoted and verified its images fine and failed only on the bake leg, with the published identity not matching the portal's. That is filed as main-cd: the bake's framework identity does not match the promoted portal image, so the publication is never sealed and every satellite is starved #3022; it will re-fire independently of this issue.

fix: a missing org grant reds the repository, not every pull request (#…

Navigation Menu

Repository navigation

CD: main `9072713` has an incomplete image set #2916

40 remaining items

Confirmed by controlled experiment, and the stall is cleared

Delivery is restored — verified at the registry, not by a green tick

What was actually wrong — and it was never the image jobs

Two things that remain open, so closing this does not bury them

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CD: main 9072713 has an incomplete image set #2916

Description

Activity

github-actions commented on Aug 31, 2026

github-actions commented on Aug 31, 2026

github-actions commented on Aug 31, 2026

github-actions commented on Aug 31, 2026

github-actions commented on Aug 31, 2026

github-actions commented on Aug 31, 2026

github-actions commented on Aug 31, 2026

github-actions commented on Aug 31, 2026

github-actions commented on Aug 31, 2026

github-actions commented on Aug 31, 2026

github-actions commented on Aug 31, 2026

github-actions commented on Sep 1, 2026

github-actions commented on Sep 1, 2026

github-actions commented on Sep 1, 2026

40 remaining items

rbuergi commented on Sep 1, 2026

Confirmed by controlled experiment, and the stall is cleared

github-actions commented on Sep 1, 2026

github-actions commented on Sep 1, 2026

github-actions commented on Sep 1, 2026

github-actions commented on Sep 1, 2026

github-actions commented on Sep 1, 2026

github-actions commented on Sep 1, 2026

github-actions commented on Sep 1, 2026

github-actions commented on Sep 1, 2026

github-actions commented on Sep 1, 2026

github-actions commented on Sep 1, 2026

rbuergi commented on Sep 2, 2026

Delivery is restored — verified at the registry, not by a green tick

What was actually wrong — and it was never the image jobs

Two things that remain open, so closing this does not bury them

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

CD: main `9072713` has an incomplete image set #2916