Skip to content

ci: run the release-path workflow guards in CI (#189) - #198

Merged
ViperJuice merged 15 commits into
mainfrom
fix/189-ci-workflow-guards
Aug 27, 2026
Merged

ViperJuice merged 15 commits into
mainfrom
fix/189-ci-workflow-guards

Conversation

@ViperJuice

@ViperJuice ViperJuice commented Aug 27, 2026 •

Copy link
Copy Markdown
Collaborator

.github/workflows/release.yml triggers only on tag push — the tag push is
the publish (build → PyPI trusted publishing → GitHub release from a CHANGELOG
awk extraction). It therefore never appears in a PR check, and every guard that
protected #188's change to it was run by hand, once. The next PR touching it
inherited none of them, on the one file where a mistake is invisible until a
release breaks.

The nastiest known mutant is silent: change the tag pattern "v*" to "V*" and
the workflow stays valid, passes actionlint, and simply never runs at tag push.
There is no red X anywhere; a version just sits tagged-but-unshipped.

Two new test.yml jobs, plus scripts/check_workflows.py.

Checked by invariant — scripts/check_workflows.py

On the parsed release.yml, whatever the diff looks like:

  • the file exists at all (outright deletion of the release path);
  • on.push.tags is exactly ["v*"];
  • publish needs build, github-release needs publish (both scalar and
    list forms of needs: accepted — the file uses the scalar);
  • publish has environment: release by name. environment: dev passes a
    presence check and ships, because GitHub auto-creates unprotected
    environments;
  • no if: and no continue-on-error on publish/github-release, at job
    level and on every step within them.
    The job-level rule alone is bypassed
    one level down: continue-on-error: true on the "Publish to PyPI" step makes
    a tag push report green while the upload failed;
  • permissions: equals the committed map exactly at both levels — job-level
    and workflow-level (absent). Checking only the job level is a concrete bypass:
    a workflow-level permissions: write-all leaves every job map untouched while
    widening build;
  • every uses: equals the committed reference exactly, so the publish action
    cannot be repointed at a fork;
  • the job set is exactly {build, publish, github-release} — an added job with
    the release environment in scope is statically decidable, so it fails here.

And across every workflow file (both .yml and .yaml): each job carries a
timeout-minutes that is an int, not a bool, in 10..30. A floor with no
ceiling reopens #187 — timeout-minutes: 360 is valid, preserves the job set,
and re-creates the six-hour default that issue exists to fix.

Checked by drift

For each changed path under .github/workflows/ (from
git diff --no-renames --diff-filter=ACMRD <base>...HEAD, which includes
deletions
— a worktree glob cannot see a file that is gone), the base job set
is compared against the head's, and any job that disappeared fails. A rename is
a removal plus an addition. A deleted file has an empty job set and is
reported, naming every job removed, rather than raising. A genuinely new
workflow file has no base version and passes.

Drift fails closed on a base that does not resolve — a shallow clone, a bad
ref, a failed fetch. Treating that the same as "the file is new" would exit 0
with all drift protection silently disabled. The base is resolved per event:
github.event.pull_request.base.sha on pull_request, github.event.before on
push, none on schedule/workflow_dispatch (invariants only, and the log
says so). github.event.before is a PushEvent field and is unset on
pull_request; wiring only that would have skipped drift on every PR.

Checked by acknowledgement

check_workflows.py is an allowlist — it catches bad states it knows how to
name. release-diff-ack is the tripwire for the remainder: any PR whose diff
touches release.yml fails unless it carries a release-change-approved label.
Labels are read live from the API, not from
github.event.pull_request.labels, because that context is frozen at trigger
time and a re-run replays the frozen payload — so "apply the label, then re-run"
actually works. It mirrors the existing changelog/skip-changelog job,
including branching on the lookup's exit status rather than on emptiness.

Explicitly NOT covered

  • A timeout above the floor but below a job's real p100 (e.g. build
    20 → 12). Not statically decidable. This is the one uncovered mutant, and it
    is in the mutation matrix with its real green exit code rather than omitted.
  • Environment protection rules, which are configured in GitHub settings and
    invisible to any file check. gh api says the release environment has
    "protection_rules": [] today, so the environment: release invariant guards
    a name, not a gate.
  • if:-skipping the guard job itself. GitHub counts a skipped job as
    satisfying a required check, so if: false on workflows merges green as
    far as branch protection is concerned. actionlint catches the literal form and
    not the equivalent non-constant one; the test job's assertions catch both,
    unless the same PR disables that job too.
  • Related: the guard runs from the PR branch, so a PR can keep the job name
    while gutting its steps. .github/CODEOWNERS is the only backstop for these,
    and required_pull_request_reviews is null today, so it gates nothing.
  • SHA-pinning pypa/gh-action-pypi-publish (currently the mutable
    @release/v1). The uses: invariant makes a change to it fail; it does not
    make it safe. Worth doing, needs its own issue.

Evidence

.consiliency/evidence/mutation-189.md. 30 mutants from
scripts/_mutate_workflow.sh, which is the single source of the set — the
loop consumes --list verbatim and a test asserts the suite exercises nothing
outside it. Mutants are committed on a scratch branch, because drift's
three-dot diff only ever sees committed changes; mutating the working tree makes
drift a silent no-op while the invariants cover for it.

  • 25 of 25 killed, each by the check meant to kill it (asserted on the reported
    reason, not just the exit code).
  • Drift shown load-bearing by stubbing job_set_drift to return []:
    maintenance-deleted, changelog-job-deleted and workflows-job-deleted all
    go green, because the release and timeout invariants only see files that
    survived.
  • The 5 expected-green cases are named individually with real exit codes.
  • actionlint 1.7.12 — the only static gate the repo had before — killed 4 of 30,
    all schema-class, and none of the ones that leave a workflow both valid and
    silent. It is pinned by archive digest (asserted before extraction) and kept
    for schema validation only.
  • release-diff-ack proven in all three directions (plus label-applied-then-rerun,
    lookup-failure fallback, and label-removed), as a local simulation of the
    shipped step body. Only the "PR does not touch release.yml" direction is live
    here — this PR does not touch release.yml.

Note for review

The uses: and permissions: allowlists are exact, so a legitimate edit to
release.yml — bumping an action, granting a scope — must update the constants
in scripts/check_workflows.py in the same PR. That is intended: on the one
workflow with no PR-time feedback, a change should be deliberate enough to be
spelled twice.

Not done here, by design: workflows and release-diff-ack still need adding
to required_status_checks.contexts via gh api after this merges. Until
then a PR that deletes the workflows job means the job never runs — drift
cannot fire, because it is what was deleted.

see #189

🤖 Generated with Claude Code

https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

ViperJuice and others added 15 commits August 26, 2026 21:48
Two DISAGREE, two PARTIALLY AGREE.

Timeout predicate: rev 1 specified '>= 5' on release.yml only. Three seats
caught it. bypass-proofs-187.md records timeout-minutes: 360 as a CLOSED bypass
that recreates the six-hour default; a floor with no ceiling reopens it, and the
floor itself had been raised 1 -> 10. Now #187's hardened predicate verbatim,
across all five workflow files.

Missing Python setup: the job was modelled on 'changelog', which runs no Python
packages, so check_workflows.py would have raised ModuleNotFoundError -- guard
dead on arrival while local uv run passed.

Self-guarding hole: GitHub runs PR jobs from the head branch's own workflow
file, so a PR deleting the guard is checked by a file that no longer defines it.
Requires branch-protection contexts + CODEOWNERS, neither a file change, plus
the if:-skip and step-gutting residuals stated plainly.

Vacuous acceptance: replacing job_set_drift with 'return []' passed all four
original criteria. Drift now has its own criteria asserting the reported reason.

Dropped the 'strictly stronger than diff-shape' claim -- six mutants pass the
invariants and actionlint. Added a label-ack tripwire for the remainder.
Verification loop now commits mutants; the working-tree loop never exercised
drift at all.

Rejected one seat's finding: actions/checkout@v7 is used repo-wide and CI is
green on it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
Rev 2's repair of rev 1 created a fresh internally-impossible contract, which
two seats caught independently: TestReleaseInvariants and AC1 demanded the
checker fail on a forked action, widened permissions and an added job, while
release_invariants() checked none of them and job_set_drift explicitly passes
additions. Those three are statically decidable, so rev 3 adds them as real
invariants (exact job set, exact permissions map, exact uses: refs) rather than
deleting the tests.

--base-ref is now resolved per event. Rev 2 named only github.event.before,
which is a PushEvent field unset on pull_request -- so every PR would have run
with an empty base and skipped drift, recreating rev 1's own WAS WRONG on the
merge path.

Drift now fails closed on an unresolvable base: a bad ref or shallow clone must
not be mistaken for a new file.

CODEOWNERS corrected -- it exists and covers these paths, but branch protection
has required_pull_request_reviews: null, so it is advisory exactly as its own
comment says. Enabling it is an operator settings decision, not a merge step.

Rejected: actions/checkout@v7 reported nonexistent; the repo uses it repo-wide
and CI is green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
Red-team seat found a real new mutant: workflow-level 'permissions: write-all'
bypasses the job-level permissions invariant entirely, since a workflow-level
block applies to any job that does not override it. Now asserted at both levels.

Also: scripts/_mutate_workflow.sh is created rather than merely referenced (the
evidence matrix could not be produced as written); actionlint is pinned to a
named version and digest with an acceptance criterion, instead of repeating the
'pinned, checksum-verified' phrasing that names neither; and release-diff-ack
gains positive tests -- rev 3 had only the negative case, which an always-fail
implementation satisfies -- plus the live-label requirement the changelog job
already documents.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
Rev 4 boarded 2 DISAGREE / 1 AGREE.

The continue-on-error invariant was job-level only. The identical mutant works
one level down: continue-on-error on the 'Publish to PyPI' STEP makes a tag push
report green while the upload failed -- the nastiest mutant in the set, still
open. The invariant now walks steps.

Deleted-file drift crashed rather than reporting. maintenance.yml deleted is not
on disk, so reading the head version raises and the checker exits with a
traceback -- which AC2 says does not count, since the reason must name the
removed jobs. The guard #187 calls the sole defence against silent job deletion
would have shipped dead. Head blob absent now means empty job set, and paths
come from git diff --diff-filter=ACMRD rather than a worktree glob, which cannot
see a file that is gone.

The mutation-helper contract was internally impossible for the fourth
consecutive revision: its set was 'exactly TestReleaseInvariants +
TestTimeoutInvariants' while the mandatory loop invoked it for two drift-only
mutants. Now an explicit union, generated from one source shared with the loop --
rev 4's loop also omitted the four mutants rev 3 and rev 4 had just added.

The actionlint criterion checked the wrong artifact: Changes verified the
archive, acceptance asserted the installed executable against the same digest.
Different byte streams; a seat confirmed the values differ.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
scripts/check_workflows.py checks release.yml invariants, timeout-minutes
across every workflow, and job-set drift against the PR base. test.yml gains
a 'workflows' job that runs it plus a pinned actionlint, and a
'release-diff-ack' job requiring a label on any PR touching release.yml.

see #189

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
…per (#189)

scripts/_mutate_workflow.sh is the single source of the mutant set: --list
emits name|expected_exit|expected_reason_tags|description, and
TestMutationHelperContract asserts the tests and the helper agree instead of
trusting that they do. The uncovered cases are listed as expected-green so the
evidence names them rather than omitting them.

see #189

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
actionlint's if-cond rule catches a literal `if: false`, so the honest
residual is the non-constant always-false form, plus gutting the checker
step's command. Both are listed as expected-green and named in the evidence.

see #189

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
…#189)

30 mutants, committed on a scratch branch so drift's three-dot diff can see
them. 25 of 25 killed by the check that is supposed to kill each; the five
expected-green cases are named individually with their real exit codes, and
drift is shown load-bearing by stubbing it to return [].

see #189

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
Both new jobs appear as check names and pass; the workflows job log shows the
drift base resolving to the real PR base sha, the actionlint archive digest
verified before extraction, and the guards exiting OK on a real runner.

see #189

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
…189)

Board on #198 found four blocking defects, all reproduced before fixing:

F1: the trigger check was an allowlist of one key, not a constraint on the
trigger set. Adding `pull_request:` or `push.branches` alongside the intact
`v*` tag filter exited 0 — making trusted publishing (id-token: write, and an
environment with no protection rules) reachable from any same-repo PR and from
every push to main. The trigger set is now exactly {push: {tags: [v*]}}.

F2: any non-zero `git diff` became an empty path list, so a base that resolves
but shares no history with HEAD ('no merge base', exit 128) passed vacuously
with drift disabled. Git failures now raise GitError and become [drift]
failures. The base-blob existence check was the same defect wearing a different
hat and is now a positive ls-tree membership test.

F3: a NEW tag-triggered workflow file was silent across all four layers. Only
release.yml and docker.yml may declare on.push.tags.

Plus: `build` joins PROTECTED_JOBS — skipping it skips publish through needs
and the tag push concludes green.

see #189

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
)

37 mutants, 31 expected-1 and all 31 killed by the credited check. The killed
table is now generated from the loop's own results file rather than
transcribed, which is what let the count read 25 against a 24-row helper.

Corrects a sentence that was false as written: release-diff-ack is the tripwire
for the remainder WITHIN release.yml, not for anything outside it — a new
tag-triggered workflow left it green, and that hole is closed by an invariant.

see #189

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
git commit-tree takes its committer from the environment and actions/checkout
sets no git identity, so the no-merge-base tests passed locally and exited 128
on the runner. Re-verified with GIT_CONFIG_GLOBAL=/dev/null and an empty HOME.

see #189

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T61zMNiPF4K29186r3uqc3
@ViperJuice
ViperJuice merged commit 8de3a33 into main Aug 27, 2026
13 checks passed
@ViperJuice
ViperJuice deleted the fix/189-ci-workflow-guards branch August 27, 2026 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant