CD: the hourly reconcile publishes a commit whose push-triggered publish is still in flight — two digests, one commit

Two CD runs published one commit as two different image sets

Core 34c3eecec (the #3366 merge, 2026-09-05 ~22:4xZ) got three Continuous Delivery (main) runs:

run	event	outcome
#7844	`workflow_run` (merge-queue completion)	cancelled
#7845	`workflow_run` (push run's completion)	promote + verify success, `Plugins: bake + seal` failure (`[FAIL] Hosting (counts unavailable — the pipeline threw before the install reported)`, the #2543 class)
#7846	`schedule` (the hourly reconcile)	promote + verify + bake + seal + register success

Both #7845 and #7846 passed decide and published: two dotnet publishes of one commit are two different digests (memex-portal-ai sha256:1e02e671… vs sha256:ebde5c56…), each tagged with its own 3.0.0-ci.<n>. Phase A's <short-sha> tag moved to the later one, so 3.0.0-ci.7845 now sits on an orphaned digest that nothing resolves to, and the 25-minute image build plus the failed bake were paid twice.

Why

decide asks ACR whether the commit's set is complete. When the reconcile tick fires while the push-triggered run is still building, the set is not complete yet, so the tick publishes too. The batching window (CD_BATCH_WINDOW_MINUTES) does not see an in-flight run either — it compares against the newest published set.

Ask

decide should treat an in-flight Continuous Delivery (main) run on the same head_sha as "being published" and skip (one REST call over actions/runs?head_sha=), the same way it already refuses to re-publish a complete set. A concurrency group keyed on the sha would do it structurally, but this workflow's concurrency is deliberately split by trigger (the reconcile lane must not be starved by push runs), so the check belongs in decide.

Observed while sealing the 3.0.0 release candidate (MeshWeaver#3355).

🤖 Generated with Claude Code

https://claude.ai/code/session_01JuXoLRvG9TfKh5mgnjGfmo

Independent occurrence 11 h earlier — same shape, and it nearly picked the wrong set

Core 9b1c58ee5 (the #3348 merge, 2026-09-05 ~11:2xZ) got three Continuous Delivery (main) runs,
and the middle one is the reconcile tick:

run	event	outcome
#7816	`workflow_run`	failed (its `decide` resolved the previous commit, `718d51f7b`)
#7817	`schedule`	promote + verify success, `Plugins: bake + seal` FAILURE
#7818	`workflow_run`	promote + verify + seal success

Both #7817 and #7818 ran Promote: tag the full set (all-or-nothing) to success — one commit,
two promotions, exactly as described. The digests differ, and I have them because I resolved both
while bumping pins:

                     ci.7817                                    ci.7818
mw-plugin-test   sha256:36da64495c67d5d9…                   sha256:dcc30bf9ee4b4fbe…
memex-portal-ai  sha256:feb8c1af91252563…                   sha256:418cec494a5102f9…

🚨 The harm is worse than a duplicated build — I nearly pinned the fleet to the losing set

ci.7817 promoted and verified its images, then failed the seal. ci.7818 sealed. Nothing in
the tag distinguishes them.

I had already resolved ci.7817's two digests and staged the Plugins pin bump onto it. What stopped
me was checking the seal job rather than the run conclusion, on a rule I happened to be carrying
("a run can conclude success and SKIP the seal"). Had I bumped, the whole fleet would have pinned
to a set whose bake never sealed — and the ordinary evidence (images promoted, Verify every image shipped green) all said go.

So this race does not merely waste 25 minutes twice: it manufactures two same-looking candidates
where one is unsealed, and the usual signals do not separate them.

A hazard in the proposed `decide` check, worth handling before it ships

"Treat an in-flight run on the same head_sha as being published and skip" has a mutual-skip failure
mode:

Self-exclusion. The querying run appears in its own actions/runs?head_sha= result. Without
filtering its own run_id it skips itself, always, and CD publishes nothing.
Simultaneous start. Two runs whose decide jobs overlap each see the other as in-flight and
both skip. Nothing publishes, and — unlike today's symptom — nothing is red, so it looks like a
quiet success. That is strictly worse than the duplicate: publishing twice is visible waste;
publishing zero times is a silent hole in delivery.

A tie-break makes it deterministic — lowest run_id (or earliest created_at) wins and proceeds,
everyone else defers — so exactly one run publishes regardless of interleaving. Whatever the shape,
the skip must be loud: a skipped publish and a not-needed publish must not look alike in the log,
which is the same trap that makes the current duplicate hard to notice.

Happy to leave the implementation with whoever owns this; flagging it because a naive version can
take delivery down quietly, and today already cost ~80 minutes of no sealed sets from a different
cause.

🔁 CD reconcile attempt 1/3 for 3a937cb (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 3a937cb. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed 0f98054 (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7847), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

✅ Healed 3a937cb (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7850), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

✅ Healed de663a7 (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7851), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

CD failed on main: run 33998613608 for commit de663a799fa36f9f8251307c8da2e9024c709ea5 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

🔁 CD reconcile attempt 1/3 for 635b2e2 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for 635b2e2. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

CD failed on main: run 34013435676 for commit 635b2e2dfeb55a00560c82a4ba565f7f49fa46a6 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

✅ Healed 635b2e2 (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7858), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

✅ Healed bbcb22f (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7859), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

🔁 CD reconcile attempt 1/3 for bbcb22f (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for bbcb22f. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

CD failed on main: run 34015096490 for commit bbcb22f256240c7132fea1e56c7df3f97564e648 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

✅ Healed bbcb22f (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7860), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

🔁 CD reconcile attempt 2/3 for bbcb22f (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for bbcb22f. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed bbcb22f (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7861), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

CD failed on main: run 34019227294 for commit bbcb22f256240c7132fea1e56c7df3f97564e648 (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

🔁 CD reconcile attempt 1/3 for ab51910 (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for ab51910. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed ab51910 (run) — main's HEAD now has the complete image set (portal 3.0.0-ci.7862), verified against ACR. Installs pick it up on the build-completion event, or within one safety-net check interval if that event never reaches them (#2494). Close this issue if nothing else is outstanding.

Fixed by #3379, merged as cc7e31d (the merge queue does not fire Closes, so closing by hand). decide now asks actions/runs?head_sha= whether an OLDER run of this workflow on the same sha is still queued/in_progress and defers with a visible decision if so — lowest run id wins, so two runs deciding at the same instant cannot both defer; a probe failure is a ::warning:: and falls open to the pre-#3379 behaviour (a duplicate set is the worst case, never a missing one). Harness cases in test-cd-steps.py: reconcile defers, push defers, nothing in flight still builds.

docs(cd): revisiting a deferral needs a CARRIER, and there is none — …

Navigation Menu

Repository navigation

CD: the hourly reconcile publishes a commit whose push-triggered publish is still in flight — two digests, one commit #3376

Two CD runs published one commit as two different image sets

Why

Ask

Independent occurrence 11 h earlier — same shape, and it nearly picked the wrong set

🚨 The harm is worse than a duplicated build — I nearly pinned the fleet to the losing set

A hazard in the proposed `decide` check, worth handling before it ships

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CD: the hourly reconcile publishes a commit whose push-triggered publish is still in flight — two digests, one commit #3376

Description

Two CD runs published one commit as two different image sets

Why

Ask

Activity

rbuergi commented on Sep 5, 2026

Independent occurrence 11 h earlier — same shape, and it nearly picked the wrong set

🚨 The harm is worse than a duplicated build — I nearly pinned the fleet to the losing set

A hazard in the proposed decide check, worth handling before it ships

github-actions commented on Sep 5, 2026

github-actions commented on Sep 5, 2026

github-actions commented on Sep 5, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 6, 2026

github-actions commented on Sep 6, 2026

rbuergi commented on Sep 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

A hazard in the proposed `decide` check, worth handling before it ships