Skip to content

A cancelled delivery run with zero jobs is a superseded queue entry — measured, documented, guarded - #3718

Merged
rbuergi merged 2 commits into
mainfrom
fix/cd-cancelled-runs-are-evicted-queue-entries
Sep 8, 2026
Merged

rbuergi merged 2 commits into
mainfrom
fix/cd-cancelled-runs-are-evicted-queue-entries

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

A cancelled delivery run with zero jobs is a superseded queue entry — measured, documented, guarded

No workflow behaviour changes. main-cd.yml's concurrency: block is byte-identical; only the comment above it moved, because it was wrong in a way that held a roll this morning.

The measurement (2026-09-08, every main-cd run created 09:00–10:09Z)

run commit conclusion jobs cancelled at next arrival, created
34207602172 3ee8dda cancelled 0 09:01:46 34207717540 @ 09:01:45
34207717540 3ee8dda cancelled 0 09:04:10 34207938426 @ 09:04:08
34207938426 3ee8dda cancelled 0 09:29:19 34210258845 @ 09:29:18
34210258845 4e1a1b6 cancelled 0 09:33:37 34210667593 @ 09:33:36
34210667593 4e1a1b6 cancelled 0 09:41:59 34211432040 @ 09:41:58
34213195672 1c61ed3 cancelled 0 10:06:05 34213625327 @ 10:06:03
34213625327 6b2fe2a cancelled 0 10:08:46 34213872077 @ 10:08:44
34206854855 765fb7f failure at its seal 25 — in flight 08:52–09:45, never cancelled
34211432040 6044910 in progress > 0 — started 09:45:52, the second the run above completed

All seven: run_attempt: 1, jobs.total_count: 0, triggering_actor: github-merge-queue[bot]. That is GitHub's one-pending-slot replacement inside our own main-cd-<ref> group (cancel-in-progress: false protects the run in flight; a waiting run is replaced by the next arrival). The commit order on main is the arrival order, so every evicted run was an ancestor of the run that replaced it — nothing was starved; the slot always held the newest commit.

What it is not, from the record alone: not the org's Actions budget (jobs never started, and MeshWeaver.Plugins runs in the same minutes executed normally); not a session or a person (seven cancellations 1–2 s after seven arrivals). What needed attention was the other conclusion in the same list — the failure on the run that did execute (that is #3461's overlap, owned elsewhere; not touched here).

What changes

  • main-cd.yml comment: the block claimed "concurrent runs QUEUE — the second waits for the first". GitHub keeps ONE waiting run and cancels the rest with zero jobs. The comment now says so, gives the reading, and states why the push lane keeps one group on the ref while dotnet-test.yml gives every main commit its own (the opposite shape, on purpose — #2412 vs #2490/#3077).
  • Docs (the durable form): ReadingCiSignals → new section "A CD run cancelled with ZERO jobs was evicted from the pending slot" — the table above, the three causes and their discriminators, what to do (nothing) and where to look instead. ModuleBuildArchitecture → "Superseded runs on main": jobs=0 alone is not the Disposal now KNOWS when it finishes: a terminal signal, and drains that cannot lie #888 starvation signature; the in-flight run beside it is.
  • Guard — MainRunsAreNeverCancelledGuard.Delivery_NeverKillsTheRunInFlight_AndKeepsTheReconcileInItsOwnLane reads main-cd.yml and evaluates (not pattern-matches) its two expressions: cancel-in-progress false for workflow_run, schedule and workflow_dispatch; the reconcile's group ≠ the push lane's; two deliveries share one group. DeliveryGuard_FiresOnEachMutation_AndStaysSilentOnTheShippedShape is the control arm: the flag flipped (literal and expression), the reconcile folded into the push lane, a per-run and a per-commit group — each fires; the shipped pair is silent.
  • What's New entry (Category: Feature).

Verified locally

  • dotnet build test/MeshWeaver.Documentation.Test -c Release -warnaserror → 0 warnings, 0 errors.
  • dotnet test … --filter MainRunsAreNeverCancelledGuard --no-build and --filter DocumentationLinkIntegrityTest --no-build → see the CI run; counts reported in the thread.
  • check-workflow-timeouts.py, check-workflow-yaml-keys.py, check-workflow-shell.py, check-pr-secret-preflight.py — self-tests and the tree, all clean.

Refs #2490, #3077, #2412. Does not touch #3461 (a peer's change set).

🤖 Generated with Claude Code

…and the lane's shape is guarded

Measured 2026-09-08 09:00–10:09Z on main-cd: seven runs (3ee8dda ×3, 4e1a1b6 ×2,
1c61ed3, 6b2fe2a) reported `cancelled` with run_attempt=1 and jobs=0, each one
1–2 s after the next CD arrival was created, while the two in-flight runs (765fb7f,
6044910) ran to their seal untouched. That is GitHub's one-pending-slot replacement
inside our own `main-cd-<ref>` group — the designed supersede for the push lane — not
an account-budget kill (jobs never started; Plugins runs in the same minutes executed)
and not a hand (seven cancellations 1–2 s after seven arrivals). It held a roll for a
morning because the workflow's own comment claimed waiting runs "QUEUE — the second
waits for the first".

- main-cd.yml: the concurrency comment now says what "queued" means in this lane, the
  reading (jobs=0, cancellation instant = next arrival's creation), and why the push
  lane keeps ONE group on the ref while Build and Test gives every main commit its own.
- ReadingCiSignals: new section with the measured table, the three causes a cancelled
  run can have and how the record alone tells them apart, and what to do (nothing —
  read the OTHER conclusion in the list, the failure on the run that executed).
- ModuleBuildArchitecture → "Superseded runs on main": jobs=0 alone is not the #888
  starvation signature; the in-flight run beside it is.
- MainRunsAreNeverCancelledGuard: Delivery_NeverKillsTheRunInFlight_AndKeepsTheReconcileInItsOwnLane
  holds main-cd's three halves (cancel-in-progress false for every event; the reconcile
  in its own group, #2490; deliveries sharing one group on the ref), with a Theory control
  arm that fires on each plausible tidy-up and stays silent on the shipped pair.
- What's New entry.

No workflow behaviour changes: the concurrency block is byte-identical; only its comment
moved. Refs #2490, #3077, #2412.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 8, 2026 10:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Minor but concrete documentation inaccuracies in the new “reading recipe” (REST endpoint/field naming) should be corrected to avoid misleading operators.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR documents and guards the intended GitHub Actions concurrency behavior where main-cd runs that are cancelled with zero jobs are superseded pending entries (not killed in-flight work), and adds tests to prevent future “tidy-ups” from breaking the delivery/reconcile lane invariants.

Changes:

  • Adds a guard test that evaluates main-cd.yml’s concurrency expressions and a control-arm matrix that must fail on plausible mutations.
  • Documents the observed “zero-jobs cancelled” signature and how to distinguish it from real cancellations (budget/manual) in CI reading docs.
  • Updates main-cd.yml commentary and adds a What’s New entry to make the behavior discoverable and durable.
File summaries
File Description
test/MeshWeaver.Documentation.Test/MainRunsAreNeverCancelledGuard.cs Adds delivery-lane concurrency guard + control-arm mutations to ensure the invariant is enforced by evaluation, not brittle matching.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-08-a-cancelled-delivery-run-with-no-jobs-is-a-superseded-queue-entry.md Adds a What’s New entry describing the “cancelled with zero jobs” interpretation and where it’s documented/guarded.
src/MeshWeaver.Documentation/Data/Architecture/ReadingCiSignals.md Adds a new section documenting the measured pattern and operational reading recipe for zero-job cancellations.
src/MeshWeaver.Documentation/Data/Architecture/ModuleBuildArchitecture.md Clarifies that jobs=0 alone is not a starvation signature for core delivery, and points to the detailed CI reading guidance.
.github/workflows/main-cd.yml Updates comments above concurrency: to describe GitHub’s one-pending-slot replacement semantics and why the lane is intentionally ref-grouped.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

# 🚨 WHAT "QUEUED" MEANS HERE — the run list will read as a defect, and it is not one. GitHub keeps
# exactly ONE PENDING run per group, and every further arrival REPLACES it: the run that was
# waiting is CANCELLED the second a newer one is created, and it reports `cancelled` having run
# ZERO jobs (`run_attempt: 1`, `jobs.total_count: 0`, `updated_at` = the next arrival's
Comment on lines +1081 to +1082
**Measured 2026-09-08, every `main-cd` run created between 09:00Z and 10:09Z** (REST:
`actions/runs/<id>` for the timestamps and attempt, `…/runs/<id>/jobs` → `total_count`):
@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

494 tests  ±0   494 ✅ ±0   43s ⏱️ -1s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 2a7bf94. ± Comparison against base commit 765cd55.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

254 tests  ±0   254 ✅ ±0   2m 10s ⏱️ -3s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 2a7bf94. ± Comparison against base commit 765cd55.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

1 073 tests  ±0   1 073 ✅ ±0   1m 26s ⏱️ -1s
    2 suites ±0       0 💤 ±0 
    2 files   ±0       0 ❌ ±0 

Results for commit 2a7bf94. ± Comparison against base commit 765cd55.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

1 666 tests  ±0   1 472 ✅ ±0   2m 13s ⏱️ +3s
    4 suites ±0     194 💤 ±0 
    4 files   ±0       0 ❌ ±0 

Results for commit 2a7bf94. ± Comparison against base commit 765cd55.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    5 files  ±0      5 suites  ±0   3m 14s ⏱️ -2s
1 703 tests ±0  1 703 ✅ ±0  0 💤 ±0  0 ❌ ±0 
1 704 runs  ±0  1 704 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 2a7bf94. ± Comparison against base commit 765cd55.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

1 191 tests  +7   1 191 ✅ +7   4m 22s ⏱️ ±0s
    3 suites ±0       0 💤 ±0 
    3 files   ±0       0 ❌ ±0 

Results for commit 2a7bf94. ± Comparison against base commit 765cd55.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Test Results

   16 files  ±0     16 suites  ±0   14m 10s ⏱️ -4s
6 381 tests +7  6 187 ✅ +7  194 💤 ±0  0 ❌ ±0 
6 382 runs  +7  6 188 ✅ +7  194 💤 ±0  0 ❌ ±0 

Results for commit 2a7bf94. ± Comparison against base commit 765cd55.

♻️ This comment has been updated with latest results.

@rbuergi
rbuergi added this pull request to the merge queue Sep 8, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 8, 2026
…nd the instrument rule

Two additions to the zero-job-cancellation section, both measured off run records:

- The arithmetic: a run holding the slot seals in ~50 min (#8103, 6044910: 09:41:58 →
  completed/success 10:31:23; #8108 started its first job 10:31:25, two seconds later, off
  the group slot). main merged ~every 7 min in that window, so every arrival that landed
  while a run held the slot was evicted by the one after it (10:01, 10:06, 10:08, 10:21,
  10:38, 10:43 — all jobs=0), and only the last arrival before the slot freed got through.
  The lever is the merge gap — a deliberate quiet window is the maintainer's call;
  cancelling other runs is not a lever.
- The instrument rule: the list endpoint's `status` is not monotonic. At 10:57Z it reported
  #8108/#8109 as `queued`, which read as a 40-minute runner wait starving every seal behind
  it; the runs' own jobs endpoint at 11:12Z said 23 jobs, 22 completed — 42 min of real
  work, no starvation. "Started nothing" is decided from jobs.total_count, started_at and
  the per-run conclusion, never from a listing's status.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rbuergi
rbuergi added this pull request to the merge queue Sep 8, 2026
Merged via the queue into main with commit f9b9a0d Sep 8, 2026
32 checks passed
@rbuergi
rbuergi deleted the fix/cd-cancelled-runs-are-evicted-queue-entries branch September 13, 2026 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants