Repository navigation
main pushes cancel each other's CI, so CD can publish nothing during a merge burst #2450
Description
The one line
.github/workflows/dotnet-test.yml:
group: build-test-${{ github.event_name == 'workflow_dispatch' && github.run_id || github.ref }}
cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}Both PR runs and main pushes group by ref. For a PR that is right — refs/pull/N/merge moves, and a superseded run is waste. For main it is not: every merge cancels the previous refs/heads/main run, and CD only fires on a Build-and-Test run that COMPLETES (workflow_run + event == 'push' && head_branch == 'main').
So while merges arrive faster than a run finishes, main produces no completed run and CD publishes nothing. The reconciler cannot paper over it either — it re-checks Consolidate test results on the commit it resolves, so it needs a completed run too.
Observed, 2026-08-26
Merges landed roughly every 10–20 minutes against a ~20-minute run:
20:30 #2433 run cancelled
20:31 #2428 run cancelled
20:38 #2354 run cancelled
20:50 #2429 run cancelled
21:11 #2328 run cancelled
21:26 b309576 ...
Newest published image stayed 4327030 (19:01) for over two hours.
Consequence, and it reached production. Plugin modules were republished 20:37–21:10 built against a newer core. With no image able to follow, memex-cloud ran ci.5876 against newer landed modules — a binary break — and every new pod aborted at boot:
Hosting failed to start
IHostedService[] -> MeshWeaver.AI.OpenAI.OpenAICompatibleModelSync
None of the constructors found … can be invoked with the available services
The rollout wedged (7 pods on the old ReplicaSet serving, new one in CrashLoopBackOff), the portal served ~11-hour-old code, and controls whose renderers live in newer view packs rendered as raw ToString() text to users. The blast-radius half is #2449; this issue is the reason the image could not catch up.
Why "just stop merging" is not the fix
It self-heals once merges pause, so it looks like an operations problem. It is not: several agent/human sessions merge independently and none can see the others' timing, so the only reliable state is "someone eventually stops", and until then image and landed modules drift apart. The failure is silent — nothing reports "CD has published nothing for two hours".
Suggested fix
Give each main commit its own concurrency group so main runs never cancel one another:
group: build-test-${{ github.event_name == 'workflow_dispatch' && github.run_id
|| github.ref == 'refs/heads/main' && github.sha
|| github.ref }}PR behaviour is unchanged (still cancel-by-ref); main commits each run to completion, so CD always has a trigger. It costs runner minutes during a burst — which is precisely what buys a publishable image.
cancel-in-progress: false for main pushes is the simpler variant, but it queues main runs serially behind each other; grouping by SHA lets them proceed in parallel.
Worth pairing with
A loud signal for "main's newest commit has no completed Build-and-Test run" / "no image published for N hours". Both times this bit, the first symptom was a user noticing production was stale — nothing upstream said so.
Activity
rbuergi commentedon Aug 26, 2026
Same defect, already fixed — PR #2447 is open with exactly this one-line change, plus a guard:
Flagging before anyone writes it twice. Structural discussion of what it does and does not solve is on #2412 (short version: #2447 makes a bad combination detectable, it does not stop it landing — that needs a merge queue, and this repo is missing the
merge_group:trigger the plugins repo already has).One mechanism correction, because it changes where you'd look
CD does fire on these runs.
main-cd.ymlsubscribes withworkflow_run: types: [completed], and a cancelled run counts as completed — so the event arrives and the workflow wakes. What it finds is noConsolidate test resultssuccess for that SHA, which is what its delivery gate actually keys on, so it decides there is nothing to publish and exits quietly.The conclusion is yours unchanged — a merge burst publishes nothing — but the distinction matters for anyone debugging this from the CD side: you will see CD runs, correctly triggered, deciding not to publish. Looking for a missing
workflow_runevent sends you hunting a trigger bug that isn't there. (I had this wrong in AGENTS.md earlier today and Copilot caught it on #2385; the corrected wording is in the AGENTS.md change that rides on #2447.)Your observation window matches
Your 10–20 minute merge cadence against a longer run time is the same thing I measured: main's five consecutive runs from 20:28–20:38 were all
cancelled, each by the next merge. The plugins repo has the identical defect with a worse consequence — itsmodulesjob publishes module bundles inside the cancelled workflow, so 18 cancelled main runs there meant bundles were never published for those commits (Plugins#729 fixes it; note it needs apush-AND-main condition rather than a ref-only one, or it breaks that repo's release lane).Closing as duplicate of #2447 unless you'd rather keep this one as the issue and link the PR to it — happy either way, just not twice.
rbuergi commentedon Aug 26, 2026
Duplicate of #2412, fixed by #2447 (
ci/main-runs-never-cancel), merged as7faac22at 21:42 UTC.I filed this after diagnosing the starvation live during the memex-cloud incident, without checking for an existing issue first — #2412 already had it. The fix that landed is broader than what I proposed here: it keeps
cancel-in-progressfor PR branches (preserving #2316's ~28% runner saving) and never cancels onmain, plus aMainRunsAreNeverCancelledGuardso the policy cannot silently regress.Confirmed working in the field within a minute of merging:
7faac22landed while667b74d's push run wasin_progress, and that run was NOT cancelled. Under the old policy it would have died, as five main runs did between 20:28 and 21:26.Two observations from the incident worth keeping, if they are not already covered by #2412:
The starvation manufactures load that prolongs it.
workflow_runfires ontypes: [completed], which includes cancelled, so every cancelled main run spawned a CD run that woke, foundgate=skipped, and no-opped. Three for667b74dalone. Withcancel-in-progress: falseon CD they queue rather than replace each other, consuming the runners needed to end the starvation.Nothing said delivery had stopped.
main-cd.ymlhas a guard for "main is GREEN and has an INCOMPLETE image set" — it cannot fire here, because main was never green: there was no completed run at all. So the observable state was a series of CD runs reporting success while publishing nothing, and the first symptom was a user noticing production was stale (~11 hours behind). A signal on "main's tip has no completed Build-and-Test" would have caught it hours earlier. Raised separately from the blast-radius half in Per-module isolation (#2234) does not cover HOSTED SERVICES — one incompatible module still kills the portal #2449.