ci: cancel superseded main runs — 28% of runner demand tested dead commits by rbuergi · Pull Request #2316 · Systemorph/MeshWeaver

rbuergi · 2026-08-26T06:39:49Z

Requested by @orwell2000 after the CI timing review. Companion to #2313 — that one fixes ~168 s of work; this one is the change that actually relieves the queue.

The measurement

concurrency.group was run_id for pushes, so every main push got its own group and no main run ever superseded another.

Over a 3.6 h window on 2026-08-25: 19 of 21 main runs were obsolete before they finished. Merges landed 1–120 seconds apart, each spawning a full ~70-runner-minute suite against a commit the next merge had already replaced — ≈1,199 runner-minutes, ~28% of all runner demand in that window, spent testing trees nothing would ever look at again.

That waste is not free, because the pool has a hard ceiling — the org is on the Team plan (60 concurrent hosted jobs) and that day's volume doubled:

day	all workflow runs	Build and Test	main pushes
20 Aug	419	103	38
24 Aug	672	131	40
25 Aug	1,092	239	63

Above the ceiling a queue hop costs 25 minutes; below it, 3 seconds. And the work did not change — this is the decisive table:

day	build	longest shard	queue (all 4 acquisitions)
20 Aug	453s	612s	8s
22 Aug	420s	612s	172s
24 Aug	457s	634s	10s
25 Aug	429s	658s	1,516s

Median run was flat at 19–20 min from 08-08 through 08-24 — straight through modularization — then 46 min on 08-25. The regression is entirely queue, so removing demand is the only thing that helps.

The change

-  group: build-test-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+  group: build-test-${{ github.event_name == 'workflow_dispatch' && github.run_id || github.ref }}
+  cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}

Both PR and push group by github.ref (refs/pull/N/merge and refs/heads/main). Only workflow_dispatch keeps run_id and never cancels — a manual dispatch is a deliberate act (it is the hand-bisect path) and must not die to an unrelated push.

Why it is safe for delivery

That is what the old comment was protecting, and each leg was checked:

The newest run is never the one cancelled — cancel-in-progress cancels the older group members. main's tip always completes, and the tip is exactly what main-cd's reconciler resolves and gates on.
main-cd no longer needs a per-commit workflow_run — its hourly reconciler (cron: 23 * * * *) asks whether main's tip has a complete image set and publishes when it does not. Publication is batched anyway (CD_BATCH_WINDOW_MINUTES), so a superseded commit was never getting its own image.
No refs/ci-green marker from a cancelled run — correct: the tree it would vouch for is one no future run will test.
No issue spam. A cancelled run still fires main-cd's workflow_run, which takes the ⛔ … nothing will be built branch — a step-summary line only. gh issue filing lives exclusively in the hourly reconcile path, so 19 cancellations/day cannot produce 19 issue comments. (Residual: a seconds-wide race where the reconciler resolves tip T and a push then cancels T's run could produce one "cannot publish" comment, self-healing on the next tick. Once an hour at worst.)

What it costs — deliberately accepted

Bisect resolution. A red main tip now covers a whole merge batch instead of one commit. Every PR is still tested as refs/pull/N/merge before merging, so what goes untested is specifically the interaction between PRs merged seconds apart — which is real. That is bisected by hand:

gh workflow run "MeshWeaver Build and Test" --ref <sha>

which is why workflow_dispatch is deliberately exempt from cancellation. The comment in the file records the exact two lines that revert this if the trade stops being worth it.

Verification

Workflow YAML parses; all 11 jobs intact; the group expression is asserted newline-free (a folded >- scalar preserves deeper-indented continuation lines literally, which would have silently changed the string — caught and avoided, and the reason it is one line).
dotnet build -c Release on MeshWeaver.Documentation.Test → 0 Error(s); the three guards that read .github/workflows/*.yml pass 14/14 on a fresh .trx.
Checked no doc or skill still asserts the old "every main push produces its own complete run" invariant — the only statement of it was the comment this PR rewrites.

No What's New entry — CI-only, no user-visible change.

Full analysis: https://claude.ai/code/artifact/427961ea-1f64-4a02-aadf-966a19b4625b

🤖 Generated with Claude Code

…mmits `concurrency.group` was `run_id` for pushes, so every main push got its own group and no main run ever superseded another. During a merge marathon that is close to pure waste. MEASURED over a 3.6 h window on 2026-08-25: 19 of 21 main runs were obsolete before they finished. Merges landed 1–120 SECONDS apart, and each one spawned a full ~70-runner-minute suite against a commit the next merge had already replaced — ≈1,199 runner-minutes, ~28% of ALL runner demand in that window, spent testing trees nothing would ever look at again. That waste is not free, because the runner pool has a hard ceiling: the org is on the Team plan (60 concurrent hosted jobs) and that day's volume doubled to 1,092 workflow runs. Above the ceiling a queue hop costs 25 MINUTES; below it, 3 seconds. Median run went 19 min (08-08 … 08-24, flat) to 46 min on 08-25 with the WORK unchanged — build 453→429 s and longest shard 612→658 s across those days. The whole regression was queue, so removing demand is the only thing that helps. Both PR and push now group by `github.ref`; only `workflow_dispatch` keeps `run_id` and never cancels, because a manual dispatch is a deliberate act (it is the hand-bisect path below) and must not die to an unrelated push. Safe for delivery, which is what the old comment was protecting: * the NEWEST run is never the one cancelled, so main's tip always completes — and the tip is exactly what main-cd's reconciler resolves and gates on; * main-cd no longer needs a per-commit workflow_run: its hourly reconciler asks whether main's tip has a complete image set and publishes if not, and publication is batched anyway, so a superseded commit was never getting its own image; * a cancelled run writes no refs/ci-green marker — correct, since the tree it would vouch for is one no future run will test; * a cancelled run still fires main-cd's workflow_run, which takes the `⛔ nothing will be built` branch: a step-summary line, no issue filed. Issue filing lives only in the hourly reconcile path, so this cannot spam. COST, deliberately accepted (maintainer, 2026-08-26): bisect resolution. A red main tip now covers a whole merge batch instead of one commit. Every PR was still tested as refs/pull/N/merge, so what goes untested is specifically the INTERACTION between PRs merged seconds apart — real, and bisected by hand with `gh workflow run "MeshWeaver Build and Test" --ref <sha>`. The comment records the exact two lines that revert this. No What's New entry: CI-only, no user-visible change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Updates the “MeshWeaver Build and Test” GitHub Actions workflow to cancel superseded in-flight runs for push to main, reducing queued runner demand during high-merge periods while preserving manual bisect capability via workflow_dispatch.

Changes:

Switches workflow-level concurrency.group to use github.ref for both PR and main push runs, so newer runs supersede older ones.
Enables cancel-in-progress for all non-workflow_dispatch events, ensuring manual dispatch runs are never cancelled.
Expands and refreshes in-file rationale documenting the tradeoff (queue relief vs. bisect granularity) and the revert lines.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-08-26T06:56:37Z