Repository navigation
ci: run the release-path workflow guards in CI (#189) - #198
Merged
Merged
Conversation
…189) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
Two DISAGREE, two PARTIALLY AGREE. Timeout predicate: rev 1 specified '>= 5' on release.yml only. Three seats caught it. bypass-proofs-187.md records timeout-minutes: 360 as a CLOSED bypass that recreates the six-hour default; a floor with no ceiling reopens it, and the floor itself had been raised 1 -> 10. Now #187's hardened predicate verbatim, across all five workflow files. Missing Python setup: the job was modelled on 'changelog', which runs no Python packages, so check_workflows.py would have raised ModuleNotFoundError -- guard dead on arrival while local uv run passed. Self-guarding hole: GitHub runs PR jobs from the head branch's own workflow file, so a PR deleting the guard is checked by a file that no longer defines it. Requires branch-protection contexts + CODEOWNERS, neither a file change, plus the if:-skip and step-gutting residuals stated plainly. Vacuous acceptance: replacing job_set_drift with 'return []' passed all four original criteria. Drift now has its own criteria asserting the reported reason. Dropped the 'strictly stronger than diff-shape' claim -- six mutants pass the invariants and actionlint. Added a label-ack tripwire for the remainder. Verification loop now commits mutants; the working-tree loop never exercised drift at all. Rejected one seat's finding: actions/checkout@v7 is used repo-wide and CI is green on it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
Rev 2's repair of rev 1 created a fresh internally-impossible contract, which two seats caught independently: TestReleaseInvariants and AC1 demanded the checker fail on a forked action, widened permissions and an added job, while release_invariants() checked none of them and job_set_drift explicitly passes additions. Those three are statically decidable, so rev 3 adds them as real invariants (exact job set, exact permissions map, exact uses: refs) rather than deleting the tests. --base-ref is now resolved per event. Rev 2 named only github.event.before, which is a PushEvent field unset on pull_request -- so every PR would have run with an empty base and skipped drift, recreating rev 1's own WAS WRONG on the merge path. Drift now fails closed on an unresolvable base: a bad ref or shallow clone must not be mistaken for a new file. CODEOWNERS corrected -- it exists and covers these paths, but branch protection has required_pull_request_reviews: null, so it is advisory exactly as its own comment says. Enabling it is an operator settings decision, not a merge step. Rejected: actions/checkout@v7 reported nonexistent; the repo uses it repo-wide and CI is green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
Red-team seat found a real new mutant: workflow-level 'permissions: write-all' bypasses the job-level permissions invariant entirely, since a workflow-level block applies to any job that does not override it. Now asserted at both levels. Also: scripts/_mutate_workflow.sh is created rather than merely referenced (the evidence matrix could not be produced as written); actionlint is pinned to a named version and digest with an acceptance criterion, instead of repeating the 'pinned, checksum-verified' phrasing that names neither; and release-diff-ack gains positive tests -- rev 3 had only the negative case, which an always-fail implementation satisfies -- plus the live-label requirement the changelog job already documents. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
Rev 4 boarded 2 DISAGREE / 1 AGREE. The continue-on-error invariant was job-level only. The identical mutant works one level down: continue-on-error on the 'Publish to PyPI' STEP makes a tag push report green while the upload failed -- the nastiest mutant in the set, still open. The invariant now walks steps. Deleted-file drift crashed rather than reporting. maintenance.yml deleted is not on disk, so reading the head version raises and the checker exits with a traceback -- which AC2 says does not count, since the reason must name the removed jobs. The guard #187 calls the sole defence against silent job deletion would have shipped dead. Head blob absent now means empty job set, and paths come from git diff --diff-filter=ACMRD rather than a worktree glob, which cannot see a file that is gone. The mutation-helper contract was internally impossible for the fourth consecutive revision: its set was 'exactly TestReleaseInvariants + TestTimeoutInvariants' while the mandatory loop invoked it for two drift-only mutants. Now an explicit union, generated from one source shared with the loop -- rev 4's loop also omitted the four mutants rev 3 and rev 4 had just added. The actionlint criterion checked the wrong artifact: Changes verified the archive, acceptance asserted the installed executable against the same digest. Different byte streams; a seat confirmed the values differ. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
scripts/check_workflows.py checks release.yml invariants, timeout-minutes across every workflow, and job-set drift against the PR base. test.yml gains a 'workflows' job that runs it plus a pinned actionlint, and a 'release-diff-ack' job requiring a label on any PR touching release.yml. see #189 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
…per (#189) scripts/_mutate_workflow.sh is the single source of the mutant set: --list emits name|expected_exit|expected_reason_tags|description, and TestMutationHelperContract asserts the tests and the helper agree instead of trusting that they do. The uncovered cases are listed as expected-green so the evidence names them rather than omitting them. see #189 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
actionlint's if-cond rule catches a literal `if: false`, so the honest residual is the non-constant always-false form, plus gutting the checker step's command. Both are listed as expected-green and named in the evidence. see #189 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
see #189 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
…#189) 30 mutants, committed on a scratch branch so drift's three-dot diff can see them. 25 of 25 killed by the check that is supposed to kill each; the five expected-green cases are named individually with their real exit codes, and drift is shown load-bearing by stubbing it to return []. see #189 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
…uards # Conflicts: # CHANGELOG.md
Both new jobs appear as check names and pass; the workflows job log shows the drift base resolving to the real PR base sha, the actionlint archive digest verified before extraction, and the guards exiting OK on a real runner. see #189 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
…189) Board on #198 found four blocking defects, all reproduced before fixing: F1: the trigger check was an allowlist of one key, not a constraint on the trigger set. Adding `pull_request:` or `push.branches` alongside the intact `v*` tag filter exited 0 — making trusted publishing (id-token: write, and an environment with no protection rules) reachable from any same-repo PR and from every push to main. The trigger set is now exactly {push: {tags: [v*]}}. F2: any non-zero `git diff` became an empty path list, so a base that resolves but shares no history with HEAD ('no merge base', exit 128) passed vacuously with drift disabled. Git failures now raise GitError and become [drift] failures. The base-blob existence check was the same defect wearing a different hat and is now a positive ls-tree membership test. F3: a NEW tag-triggered workflow file was silent across all four layers. Only release.yml and docker.yml may declare on.push.tags. Plus: `build` joins PROTECTED_JOBS — skipping it skips publish through needs and the tag push concludes green. see #189 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
) 37 mutants, 31 expected-1 and all 31 killed by the credited check. The killed table is now generated from the loop's own results file rather than transcribed, which is what let the count read 25 against a 24-row helper. Corrects a sentence that was false as written: release-diff-ack is the tripwire for the remainder WITHIN release.yml, not for anything outside it — a new tag-triggered workflow left it green, and that hole is closed by an invariant. see #189 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
git commit-tree takes its committer from the environment and actions/checkout sets no git identity, so the no-merge-base tests passed locally and exited 128 on the runner. Re-verified with GIT_CONFIG_GLOBAL=/dev/null and an empty HOME. see #189 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
This was referenced Aug 27, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
.github/workflows/release.ymltriggers only on tag push — the tag push isthe publish (build → PyPI trusted publishing → GitHub release from a CHANGELOG
awk extraction). It therefore never appears in a PR check, and every guard that
protected #188's change to it was run by hand, once. The next PR touching it
inherited none of them, on the one file where a mistake is invisible until a
release breaks.
The nastiest known mutant is silent: change the tag pattern
"v*"to"V*"andthe workflow stays valid, passes actionlint, and simply never runs at tag push.
There is no red X anywhere; a version just sits tagged-but-unshipped.
Two new
test.ymljobs, plusscripts/check_workflows.py.Checked by invariant —
scripts/check_workflows.pyOn the parsed
release.yml, whatever the diff looks like:on.push.tagsis exactly["v*"];publishneedsbuild,github-releaseneedspublish(both scalar andlist forms of
needs:accepted — the file uses the scalar);publishhasenvironment: releaseby name.environment: devpasses apresence check and ships, because GitHub auto-creates unprotected
environments;
if:and nocontinue-on-erroronpublish/github-release, at joblevel and on every step within them. The job-level rule alone is bypassed
one level down:
continue-on-error: trueon the "Publish to PyPI" step makesa tag push report green while the upload failed;
permissions:equals the committed map exactly at both levels — job-leveland workflow-level (absent). Checking only the job level is a concrete bypass:
a workflow-level
permissions: write-allleaves every job map untouched whilewidening
build;uses:equals the committed reference exactly, so the publish actioncannot be repointed at a fork;
{build, publish, github-release}— an added job withthe release environment in scope is statically decidable, so it fails here.
And across every workflow file (both
.ymland.yaml): each job carries atimeout-minutesthat is anint, not abool, in10..30. A floor with noceiling reopens #187 —
timeout-minutes: 360is valid, preserves the job set,and re-creates the six-hour default that issue exists to fix.
Checked by drift
For each changed path under
.github/workflows/(fromgit diff --no-renames --diff-filter=ACMRD <base>...HEAD, which includesdeletions — a worktree glob cannot see a file that is gone), the base job set
is compared against the head's, and any job that disappeared fails. A rename is
a removal plus an addition. A deleted file has an empty job set and is
reported, naming every job removed, rather than raising. A genuinely new
workflow file has no base version and passes.
Drift fails closed on a base that does not resolve — a shallow clone, a bad
ref, a failed fetch. Treating that the same as "the file is new" would exit 0
with all drift protection silently disabled. The base is resolved per event:
github.event.pull_request.base.shaonpull_request,github.event.beforeonpush, none onschedule/workflow_dispatch(invariants only, and the logsays so).
github.event.beforeis a PushEvent field and is unset onpull_request; wiring only that would have skipped drift on every PR.Checked by acknowledgement
check_workflows.pyis an allowlist — it catches bad states it knows how toname.
release-diff-ackis the tripwire for the remainder: any PR whose difftouches
release.ymlfails unless it carries arelease-change-approvedlabel.Labels are read live from the API, not from
github.event.pull_request.labels, because that context is frozen at triggertime and a re-run replays the frozen payload — so "apply the label, then re-run"
actually works. It mirrors the existing
changelog/skip-changelogjob,including branching on the lookup's exit status rather than on emptiness.
Explicitly NOT covered
build20 → 12). Not statically decidable. This is the one uncovered mutant, and it
is in the mutation matrix with its real green exit code rather than omitted.
invisible to any file check.
gh apisays thereleaseenvironment has"protection_rules": []today, so theenvironment: releaseinvariant guardsa name, not a gate.
if:-skipping the guard job itself. GitHub counts a skipped job assatisfying a required check, so
if: falseonworkflowsmerges green asfar as branch protection is concerned. actionlint catches the literal form and
not the equivalent non-constant one; the
testjob's assertions catch both,unless the same PR disables that job too.
while gutting its steps.
.github/CODEOWNERSis the only backstop for these,and
required_pull_request_reviewsisnulltoday, so it gates nothing.pypa/gh-action-pypi-publish(currently the mutable@release/v1). Theuses:invariant makes a change to it fail; it does notmake it safe. Worth doing, needs its own issue.
Evidence
.consiliency/evidence/mutation-189.md. 30 mutants fromscripts/_mutate_workflow.sh, which is the single source of the set — theloop consumes
--listverbatim and a test asserts the suite exercises nothingoutside it. Mutants are committed on a scratch branch, because drift's
three-dot diff only ever sees committed changes; mutating the working tree makes
drift a silent no-op while the invariants cover for it.
reason, not just the exit code).
job_set_drifttoreturn []:maintenance-deleted,changelog-job-deletedandworkflows-job-deletedallgo green, because the release and timeout invariants only see files that
survived.
all schema-class, and none of the ones that leave a workflow both valid and
silent. It is pinned by archive digest (asserted before extraction) and kept
for schema validation only.
release-diff-ackproven in all three directions (plus label-applied-then-rerun,lookup-failure fallback, and label-removed), as a local simulation of the
shipped step body. Only the "PR does not touch release.yml" direction is live
here — this PR does not touch
release.yml.Note for review
The
uses:andpermissions:allowlists are exact, so a legitimate edit torelease.yml— bumping an action, granting a scope — must update the constantsin
scripts/check_workflows.pyin the same PR. That is intended: on the oneworkflow with no PR-time feedback, a change should be deliberate enough to be
spelled twice.
Not done here, by design:
workflowsandrelease-diff-ackstill need addingto
required_status_checks.contextsviagh apiafter this merges. Untilthen a PR that deletes the
workflowsjob means the job never runs — driftcannot fire, because it is what was deleted.
see #189
🤖 Generated with Claude Code
https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.