A cancelled delivery run with zero jobs is a superseded queue entry — measured, documented, guarded

.github/workflows/main-cd.yml

@@ -147,25 +147,39 @@ permissions:
   # MeshWeaver.Plugins CI can pull it with a GitHub token instead of ACR credentials.
   packages: write
 
-# Every green main merge ships its OWN image increment — CD runs never supersede each
-# other. The previous shared `main-cd` group dated from when this workflow also ROLLED
-# OUT deployments; delivery is pull-based now (installs self-update by comparing the
-# monotonic 3.0.0-ci.<n> version tag), so parallel image builds are safe: every tag is
-# distinct and nothing here mutates a live environment. Under the shared group GitHub's
-# one-pending-slot rule CANCELLED queued CD runs during merge bursts (6 cancelled runs
-# on 2026-08-07 alone), silently dropping release increments.
-# 🚨 SERIALIZE, and never cancel. The group used to be keyed on `github.run_id`, which is unique
-# per run — so it grouped each run with itself and serialized NOTHING. Two CD runs for the same
-# commit could therefore overlap, and because each one skipped when it saw "another CD run is
-# already in flight", BOTH deferred and NEITHER published. Observed 2026-08-19 on cb33558: a
-# workflow_run CD and a manual dispatch raced, both reported success, and no image was built —
-# a mutual deferral that reads exactly like a healthy skip.
+# 🚨 SERIALIZE, and never cancel the run in flight. The group used to be keyed on `github.run_id`,
+# which is unique per run — so it grouped each run with itself and serialized NOTHING. Two CD runs
+# for the same commit could therefore overlap, and because each one skipped when it saw "another
+# CD run is already in flight", BOTH deferred and NEITHER published. Observed 2026-08-19 on
+# cb33558: a workflow_run CD and a manual dispatch raced, both reported success, and no image was
+# built — a mutual deferral that reads exactly like a healthy skip. Keyed on the ref, ONE push-lane
+# run executes at a time, and `cancel-in-progress: false` matters as much as the group — a CD run
+# midway through pushing a multi-image set or sealing a publication must never be killed, or the
+# set is left incomplete, which is the very state the reconciler exists to repair.
 #
-# Keyed on the ref, concurrent runs QUEUE instead: the second waits for the first, then finds a
-# complete image set and exits with "nothing to reconcile", which is the honest no-op. And
-# `cancel-in-progress: false` matters as much as the group — a CD run that is midway through
-# pushing a multi-image set must never be killed, or the set is left incomplete, which is the
-# very state the reconciler exists to repair.
+# 🚨 WHAT "QUEUED" MEANS HERE — the run list will read as a defect, and it is not one. GitHub keeps
+# exactly ONE PENDING run per group, and every further arrival REPLACES it: the run that was
+# waiting is CANCELLED the second a newer one is created, and it reports `cancelled` having run
+# ZERO jobs (`run_attempt: 1`, `jobs.total_count: 0`, `updated_at` = the next arrival's
+# `created_at` ± 2 s). The run in flight is never touched. That is the supersede rule for this lane,
+# applied by GitHub in arrival order: a delivery for an older commit that has not started is worth
+# nothing once a newer commit's run is waiting — the newer run builds a superset — and if the push
+# lane ever falls behind, the hourly reconcile targets the newest commit CI has vouched for
+# (#3077). Measured 2026-09-08 09:00–10:09Z: seven such runs (3ee8dda6e ×3, 4e1a1b633 ×2,
+# 1c61ed3ae, 6b2fe2a10) were cancelled 1–2 s after the next arrival was created, jobs=0, while the
+# in-flight runs 765fb7f52 and 60449106f ran to their seal untouched — and the reading "CD keeps
+# getting killed" still held a roll for a morning. A zero-job cancellation cannot have torn a set
+# or a seal; the conclusion to read is the FAILURE on the run that did execute. Reading recipe, and
+# how to tell this from an account-budget kill or a hand:
+#   Doc/Architecture/ReadingCiSignals → "A CD run cancelled with ZERO jobs"
+#
+# Why NOT a per-commit group, the shape dotnet-test.yml uses so a main test run is never evicted
+# (MainRunsAreNeverCancelledGuard holds both shapes, deliberately opposite): a test run for a landed
+# commit is evidence nothing else produces, so evicting it loses the only build of that tree. A
+# delivery run for a superseded commit produces images nobody will pull and a publication the next
+# run re-seals — parallel image builds would be safe (every tag is distinct, delivery is pull-based),
+# but two CD runs publishing the SAME framework identity at once is the overlap #3461 describes,
+# and a per-commit push lane would add that writer on every merge burst.
 concurrency:
   # 🚨 The RECONCILE (schedule) has its own lane. It is the delivery path — dotnet-test.yml says
   # so where it justifies cancel-in-progress on main — and on 2026-08-27 it was STARVED by the

src/MeshWeaver.Documentation/Data/Architecture/ModuleBuildArchitecture.md

@@ -405,6 +405,15 @@ run publishing between a newer run's `select` and its hand-over.
      22 of the last 25 main runs were cancelled with jobs=0."* **Nothing published for hours.**
      Protecting past-gates runs is what stops that: in a burst, every run that proves green still
      seals, so the fleet gets a publication roughly every gate-suite length instead of none.
+   * 🚨 **`jobs=0` alone is NOT the starvation signature — read the in-flight run beside it.**
+     Core's own `main-cd` keeps ONE group on the ref with `cancel-in-progress: false`, so GitHub
+     applies both guards itself: the run in flight (past its gates by construction) is never
+     touched, and a run still *waiting* is replaced by the next arrival, reporting `cancelled`
+     with zero jobs. Measured 2026-09-08: seven such evictions in an hour, both in-flight runs
+     sealed, nothing starved — the slot always held the newest commit. What GitHub does not check
+     is ancestry, and the hourly reconcile bounds that to one tick. Reading a zero-job
+     cancellation as #888 without an in-flight run that was killed is the misread to avoid:
+     [Reading CI Signals](../ReadingCiSignals) → "A CD run cancelled with ZERO jobs".
 2. **Only `push` → `main` supersedes `push` → `main`.** A `repository_dispatch` (the platform wave)
    and the `schedule` poll are never cancelled by this rule and never cancel anything — the #826
    rule, kept intact. Each `main` push keeps its own concurrency group keyed on the commit, so GitHub

src/MeshWeaver.Documentation/Data/Architecture/ReadingCiSignals.md

@@ -1068,6 +1068,84 @@ nothing. The two together are what makes off-branch triggers destructive.
 Guarded by `WorkflowRunTriggerBranchFilterGuard`, which carries a control arm: if its block matcher
 ever stops recognising `workflow_run:`, it fails rather than passing having examined nothing.
 
+### 🚨 A CD run cancelled with ZERO jobs was evicted from the pending slot — the delivery it superseded is the run still in flight
+
+The section above is the *unfiltered* trigger, fixed on 2026-09-02. What remains — by design — is
+GitHub's own rule inside the filtered group: **one run executes, one waits, and every further
+arrival REPLACES the one waiting.** The replaced run reports `cancelled` with `run_attempt: 1` and
+**zero jobs**; the run in flight is untouched (`cancel-in-progress: false`). Read as "CD is being
+killed", this shape has now held a roll twice — 2026-08-30 on Build and Test (#2412, where it WAS a
+defect and got a per-commit group) and 2026-09-08 on `main-cd` (where it is the design) — each time
+while the run that mattered was executing normally.
+
+**Measured 2026-09-08, every `main-cd` run created between 09:00Z and 10:09Z** (REST:
+`actions/runs/<id>` for the timestamps and attempt, `…/runs/<id>/jobs` → `total_count`):
+
+| run | commit | conclusion | jobs | cancelled at | next arrival, created |
+|---|---|---|---|---|---|
+| 34207602172 | 3ee8dda6e | cancelled | 0 | 09:01:46 | 34207717540 @ 09:01:45 |
+| 34207717540 | 3ee8dda6e | cancelled | 0 | 09:04:10 | 34207938426 @ 09:04:08 |
+| 34207938426 | 3ee8dda6e | cancelled | 0 | 09:29:19 | 34210258845 @ 09:29:18 |
+| 34210258845 | 4e1a1b633 | cancelled | 0 | 09:33:37 | 34210667593 @ 09:33:36 |
+| 34210667593 | 4e1a1b633 | cancelled | 0 | 09:41:59 | 34211432040 @ 09:41:58 |
+| 34213195672 | 1c61ed3ae | cancelled | 0 | 10:06:05 | 34213625327 @ 10:06:03 |
+| 34213625327 | 6b2fe2a10 | cancelled | 0 | 10:08:46 | 34213872077 @ 10:08:44 |
+| 34206854855 | 765fb7f52 | **failure**, at its seal | 25 | — | in flight 08:52–09:45, never cancelled |
+| 34211432040 | 60449106f | in progress | > 0 | — | started 09:45:52, the second the run above completed |
+
+Seven cancellations, seven arrivals, each 1–2 s apart, `run_attempt: 1` on every one; the two runs
+that were executing ran to their seal. Nothing was starved: the pending slot always held the
+**newest** commit, and the commit order on `main` (3ee8dda6e → 4e1a1b633 → 60449106f → 1c61ed3ae →
+6b2fe2a10) is the arrival order, so every evicted run was an ancestor of the run that replaced it.
+(Three runs for one commit is the merge queue: Build and Test completes on the queue branch and on
+`main` for the same sha, and each completion is an arrival — no-ops that still hold the slot,
+#2490.)
+
+**The arithmetic under continuous merging.** A run holding the slot seals in roughly 50 minutes
+(#8103, 60449106f: created 09:41:58, `completed/success` 10:31:23; the next run, #8108, started its
+first job at 10:31:25 — two seconds later, off the group slot, not a runner). In the same window
+`main` merged about every seven minutes, so every arrival that landed while a run held the slot was
+evicted by the one after it (10:01, 10:06, 10:08, 10:21, 10:38, 10:43 — all `jobs=0`), and only an
+arrival that happened to be the LAST before the slot freed got through. The commit that ships is
+therefore "newest at the moment the slot frees", and a commit that lands in the middle of a burst
+is delivered only as part of a later run's superset — never as its own. **The lever is the merge
+gap**, not the workflow: a deliberate merge-quiet window when a specific commit must seal is the
+maintainer's call, and cancelling other runs to make room is not a lever at all — the run in flight
+is the one doing the delivering.
+
+**The three things a cancelled run can be, told apart from the record alone:**
+
+| cause | `jobs` | timing | corroboration |
+|---|---|---|---|
+| evicted from the pending slot (this shape) | **0** | `updated_at` = the next arrival's `created_at` ± 2 s | the in-flight run of the same group is untouched |
+| the org's Actions budget or job ceiling | > 0, jobs cut mid-flight | no correlation with arrivals; runs in **every** repository of the org stop in the same minutes | a billing banner on the org — and here, `MeshWeaver.Plugins` runs in the same minutes executed normally |
+| a session or a person | any | no correlation with arrivals | seven cancellations 1–2 s after seven arrivals is not a hand |
+
+🚨 **The instrument rule — decide "started nothing" from the run's OWN record, never from the list's
+`status`.** The list endpoint's `status` is not monotonic: at 10:57Z it reported #8108 and #8109
+(both 765cd55e6, created 10:30) as `queued`, which read as "waiting 40 minutes for a runner — the
+in-flight slot is starving every seal behind it, the #888 shape by another mechanism". The runs'
+own `jobs` endpoint at 11:12Z said 23 jobs, 23 started, 22 completed — 42 minutes of real work
+against a 50-minute bake, no starvation, and a retraction. `jobs.total_count`, each job's
+`started_at`, and the per-run `conclusion` are the readings; `queued` on a listing is not one.
+
+**What to do with one: nothing.** The commit it would have built is an ancestor of the commit the
+slot now holds, and the push lane builds that one next; if the push lane ever falls behind, the
+hourly reconcile targets the newest commit CI has vouched for (#3077). A cancelled run with zero
+jobs cannot have torn an image set or a publication — it never ran a step. **The conclusion that
+matters is the OTHER one in the same list**: a `failure` on a run that DID execute (34206854855
+above, red at its seal), which no amount of "the runs keep getting cancelled" explains.
+
+**Why `main-cd` keeps ONE group on the ref while Build and Test gives every main commit its own** —
+`MainRunsAreNeverCancelledGuard` holds both shapes, deliberately opposite, each with a control arm
+that fires on the other's expression: a test run for a landed commit is evidence nothing else
+produces, so evicting it loses the only build of that tree; a delivery run for a superseded commit
+produces images nobody will pull and a publication the next run re-seals, and two CD runs
+publishing the same framework identity at once is the overlap #3461 describes. So on `main-cd` the
+eviction IS the supersede rule, applied by GitHub in arrival order. The one thing GitHub does not
+check is ancestry: if Build and Test ever completed out of commit order, an older commit's run
+could evict a newer one's — and the reconcile's next tick covers that hour.
+
 ### The other direction: a CD red on main that means nothing
 
 The section above is delivery stopping behind green ticks. The inverse cost the same evening: the

src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-08-a-cancelled-delivery-run-with-no-jobs-is-a-superseded-queue-entry.md

@@ -0,0 +1,34 @@
+---
+Name: A cancelled delivery run with no jobs is a superseded queue entry
+Category: Feature
+Description: When merges land faster than delivery runs finish, the runs list fills with "cancelled" delivery runs. Each of those had zero jobs and was replaced by a newer commit's run before it started — the run that was executing was never touched. The reading, and the three lines that keep a live delivery alive, are now written down and guarded.
+Icon: DocumentBulletList
+Order: -20260908
+---
+
+During a merge burst the delivery workflow's run list looks alarming: run after run marked
+**cancelled**, one per merge. Twice now that reading has held a roll — once when it was a real
+defect, and once, on 2026-09-08, when it was not.
+
+## What was actually happening
+
+GitHub keeps exactly one *waiting* run per concurrency group. When a newer commit's delivery run
+arrives while one is already waiting, the waiting one is cancelled and the newer one takes its
+place. The cancelled run never ran a single job. The run that was **executing** — building images,
+sealing a publication — is never touched.
+
+Measured over one hour: seven delivery runs cancelled, each one to two seconds after the next
+arrival was created, all with zero jobs; the two runs that were executing ran to their seal. The
+newest commit always held the slot, so nothing was starved. What needed attention was a different
+conclusion in the same list — a *failure* on a run that did execute.
+
+## What is now written down and held
+
+- [Reading CI Signals](/Doc/Architecture/ReadingCiSignals) carries the measured table, the reading
+  recipe (`jobs: 0`, and the cancellation instant equal to the next arrival's creation), and how to
+  tell this apart from an account-budget kill or a hand.
+- The delivery workflow's own comment says what "queued" means in this lane, instead of claiming
+  that waiting runs queue up behind each other.
+- A guard test holds the three lines the shape depends on: the run in flight is never cancelled for
+  any event, the hourly reconcile keeps its own lane, and deliveries share one group on the ref —
+  with a control arm that fires on each plausible "tidy-up".