The hourly CD reconcile is STARVED by no-op workflow_run runs — the cron missed 07:23 and 08:23 while six no-op runs occupied its slot #2490

Activity

1. rbuergi

   closed this as completedin #2491on Aug 27, 2026
2. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   ContributorAuthor

   More actions

   Recurrence 2026-08-30 — same starvation, different driver: MERGE VELOCITY, not no-op runs

   This closed as the reconcile lane being starved by no-op workflow_run runs holding the group's single pending slot, and the fix keyed the schedule into its own lane. The push lane still starves itself, and it did so today for ~45 minutes.

   Measured 11:13Z

   • last Continuous Delivery (main) success: 33306532232, sha de321a440, 10:27:41Z;
   • of the last 60 runs: 42 cancelled, 14 success, 2 failure, 2 pending;
   • five CD runs exist for the single sha 9003d0d0f — four cancelled, one pending.

   Mechanism — unchanged, and it is the same one this issue names

   concurrency:
     group: main-cd-${{ github.event_name == 'schedule' && 'reconcile' || github.ref }}
     cancel-in-progress: false

   cancel-in-progress: false correctly protects a run that is mid-publish (an incomplete image set is the state the reconciler exists to repair). But GitHub keeps only one pending slot per group, so each new CD arrival evicts the previously queued one. Every merge to main fires a Build-and-Test, every success fires a CD — and with an ~18-minute multi-arch publish, CD completes only when merges happen to pause.

   The 2026-08-27 driver was six no-op runs on one sha. Today's driver is ordinary merge traffic: two agent sessions landing PRs on green, which is exactly what the pull-based self-update wants them to do. Nothing is misbehaving; the lane simply cannot absorb a normal busy day. That is why this is worth reopening rather than filing as a new flake — the shape this issue fixed for schedule is still live for push.

   Why nothing was changed here

   The current concurrency block IS this issue's fix, and improvising a third variant under time pressure is how the same bug gets a third form. The immediate mitigation was social and is in place: both agent sessions are holding core merges so CD gets a clear window (AGENTS.md — "merging is a shared action … check whether main has a run someone is gating a deploy on; if so, hold and say so").

   The design question, for whoever picks this up

   A queue that evicts is the wrong primitive for a delivery lane where only the newest sha matters. Two shapes worth weighing, neither obviously right:

   1. Let it cancel. cancel-in-progress: true plus the existing all-or-nothing staging-tag promotion — a cancelled run publishes no consumer-visible tag by construction (staging-<sha>-<run_id> is unselectable), so the objection in the header ("a run midway through pushing a multi-image set must never be killed") may already be answered by the promotion design that landed after this comment was written. Worth re-reading that guarantee before trusting it.
   2. Coalesce instead of queue. One run per settle-window rather than one per merge — CD only ever needs the newest sha, and today it built four supersedable runs for one.

   Either way the property to hold is: a busy hour must not mean zero deliveries.
3. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   ContributorAuthor

   More actions

   Second half of this defect, measured 2026-08-30 12:35Z: a "green" CD run can be a no-op, and today most of them are

   My earlier comment described the eviction half — the one-pending-slot rule discarding queued deliveries. That is real but it is not why nothing is shipping. The lane is also reporting success while publishing nothing, and that is far worse, because eviction is visible and this is not.

   Measured

   Three CD runs on sha 0c584a2d5, all completed/success:

   33311469979  success   0c584a2d5
   33311337339  success   0c584a2d5
   33311280959  success   0c584a2d5
   

   Jobs of 33311469979:

   Resolve the target commit and decide whether to publish : skipped
   Build + push portal-ai image                            : skipped
   Build + push migration image                            : skipped
   Notify platform-update registry                         : skipped
   Plugins: resolve this run's tester image digest         : skipped
   Promote: tag the full set (all-or-nothing)              : skipped
   

   Same shape on 33308670536, where the only job that executed is the one named "CD delivered, or had a good reason not to" — which passes on the second branch. ACR's newest set is still be39fc6 / ci.6691; there is no image for 0c584a2d5 despite three green runs.

   Mechanism

   CD is workflow_run on MeshWeaver Build and Test completing successfully for a push to main. On a busy trunk every merge moves main and cancels the in-flight Build-and-Test. A cancelled run does not satisfy the filter, so CD's if skips every job — and a workflow whose jobs are all skipped concludes success.

   So the faster main moves, the more reliably CD reports green and ships nothing. The two halves compound: eviction removes the queued real runs, and cancellation converts the survivors into green no-ops.

   Why this is the dangerous half

   "CD is green" is the signal humans and agents use to decide the platform is delivering. Today it was false for at least four consecutive runs, and I personally reported "CD drained, current main is delivered" to a maintainer on the strength of a run conclusion — twice — before checking the jobs. The eviction problem announces itself (cancelled is visible in the list); this one is indistinguishable from health without opening a run and reading job outcomes.

   This is the same class as SKIPPED/absent required contexts reading as satisfied, documented in Reading CI Signals — but on the delivery lane rather than the test lane.

   What would fix the signal (cheap, and separable from the queue design)

   The run must not be able to conclude success having published nothing. CD delivered, or had a good reason not to should assert the reason: if the trigger run was cancelled, that is not a good reason to skip — it is a delivery that was silently dropped, and it should fail loudly or re-target the current head. A no-op that reports success is precisely the gate-that-cannot-fail shape this repo already refuses everywhere else.

   And the queue design, restated

   The trigger should be resilient to a cancelled predecessor at all — CD only ever needs to deliver the newest sha, so "the run that would have delivered was cancelled" should re-target rather than skip. Coalescing on a settle-window (one run per quiet period, always against current head) satisfies both halves at once.

   Interim mitigation in force: both agent sessions are spacing merges to give main a quiet window in which one Build-and-Test can survive uncancelled.
4. rbuergi commented on Aug 30, 2026

   

   rbuergi

   on Aug 30, 2026

   ContributorAuthor

   More actions

   Correction to my own method above — and a better one

   The finding in my previous comment stands: a CD run can conclude success with every publishing job skipped, and several did today. But the method I gave for detecting it is subtly wrong, and I proved it wrong on myself within the hour.

   What I did: filtered CD runs to conclusion == "success", then inspected each one's Promote job. On that basis I concluded nothing had published since 11:26Z.

   What was actually true: run 33311588750 had already promoted:

   run status = in_progress    conclusion = null    head_sha = 0c584a2d5
     Promote: tag the full set (all-or-nothing) : success
     Verify every image shipped                 : success
     Notify platform-update registry            : success
   

   ACR confirms the whole set shares one digest — 3.0.0-rc8.ci.6714, 0c584a2, main and staging-0c584a2-33311588750 all sha256:6d6fda2dcb6ab.

   So the run that published was excluded by my filter because it had not finished. A CD run promotes early and keeps going (bake/wave legs run after), so "has it published" and "has the run concluded" are independent questions — and I had just finished arguing that the run conclusion does not answer the first one, before using it to decide what to look at.

   The reliable method, in order of preference

   1. Ask the registry, and let the artifact name the run. The staging tag is literally staging-<sha>-<run_id>, so the digest set identifies its own producer with no scanning at all:
      • list tags, group by digest — a promoted set shares one digest across the version tag, the commit tag and main;
      • read the run id straight out of the staging-… tag.
   2. If you must scan Actions: inspect the Promote job of every CD run in the window, regardless of run conclusion — in_progress, success, all of them. Never filter by run conclusion first.

   The generalisation, since this is now three-for-three today

   Today produced three guards that asserted the wrong thing and were green while the property failed: a bridge guard matching a spelling rather than the construct; MainRunsAreNeverCancelledGuard asserting the cancel-in-progress setting while runs were being evicted; and a ToTask ratchet budgeting 1,548 allowances against an actual 2. My scan above is the fourth, in the reader rather than in a gate.

   A check that asserts configuration, spelling, or status cannot see the outcome failing. Assert the outcome, under the inputs that would break it, and prove it by mutation.

   Filed here rather than only in PR descriptions, because the CD lane is where believing a green signal costs the most.