Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,15 @@ jobs:
score-push: true
```

The action also verifies your [workflow lockfile](https://docs.github.com/en/actions/concepts/security/workflow-dependency-lockfiles) (`.github/workflows/actions.lock`) with `gh actions-lock --verify`, and fails the job if it doesn't verify (`soft-fail` doesn't change that). A repository without a lockfile gets a warning and the job continues. Create it from the repository root and commit it:

```sh
gh extension install github/gh-actions-lock
gh actions-lock
```

Set `verify-lockfile: false` to opt out.

**Full guide:** [getplumber.io/docs/cli/github#run-with-github-actions](https://getplumber.io/docs/cli/github#run-with-github-actions)

## GitLab CI component
Expand Down
74 changes: 74 additions & 0 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ inputs:
verify-attestation:
description: "Verify the downloaded binary's build-provenance attestation (sigstore/SLSA) against the getplumber/plumber release workflow, using the gh CLI. This anchors the binary to a trusted build regardless of the mutable release tag. Disable for air-gapped or GHES setups without attestation access."
default: "true"
verify-lockfile:
description: "Verify the repository's workflow lockfile (.github/workflows/actions.lock) with 'gh actions-lock --verify': every locked action commit must still come from the repository and ref it names. On by default. A repository without a lockfile gets a warning and the job continues; create one with 'gh extension install github/gh-actions-lock' then 'gh actions-lock' from the repository root, and commit it. A failed verification fails the job, and soft-fail does not mask it. Skipped with a warning when project is set (no checkout to verify) or on GitHub Enterprise Server (not supported by gh-actions-lock). Set false to opt out."
default: "true"
actions-lock-version:
description: "gh-actions-lock release used by verify-lockfile, without the leading v (e.g. 0.4.1)."
default: "0.4.1"
github-token:
description: "Token for the GitHub API (branch protection, advisory DB, workflow states, runs, releases) and SARIF upload. Needs Administration:read for full branchMustBeProtected, actions:read to resolve a job's claims (whether it can run, and the repository default token's real write) from workflow states, the latest runs and one run's job log, and the job needs security-events:write to upload SARIF. A refused read on any of these is never an error: the claim it would have settled stays unknown."
default: ${{ github.token }}
Expand Down Expand Up @@ -170,6 +176,56 @@ runs:
chmod +x "${bindir}/plumber${ext}"
echo "$bindir" >> "$GITHUB_PATH"

# Verifies .github/workflows/actions.lock. A missing lockfile only warns.
# This step never fails the job itself: it records a status and the
# Enforce gate step fails the job on a failed verification, after SARIF
# and artifacts are uploaded. soft-fail does not mask that.
- name: Verify workflow lockfile

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Test coverage

Medium: the new verify-lockfile step and the enforce-gate lockfile precedence are entirely untested, so a regression that masks a supply-chain failure (e.g. soft-fail swallowing a missing/failed lockfile) would ship silently. (Medium)

This PR adds the whole workflow-lockfile feature — the Verify workflow lockfile step (action.yml:182), its status propagation into the summary (action.yml:104-111), and the precedence branch in Enforce gate (action.yml:385-392) — but the only test added (scripts/release-bump-version.test.sh:47) just checks that the actions-lock-version default is left alone during a release bump. The new decision logic has no test at all.

The repo already has a harness for exactly this kind of shell logic: scripts-test.yml runs scripts/*.test.sh and triggers on action.yml changes, and cmd/action_template_test.go pins action.yml shape from Go. The new logic is inline in the run: blocks, so nothing exercises it.

The most important untested guarantee is the one the code documents at action.yml:385-387: "soft-fail does not mask it." That ordering (the LOCKFILE_STATUS case before the EXIT_CODE/SOFT_FAIL handling) is the entire point of the feature and is load-bearing for supply-chain security. If a future edit reorders those blocks, or moves the lockfile case into the code==1 branch, a run with soft-fail: true, EXIT_CODE=0, LOCKFILE_STATUS=missing would exit 0 and the job would pass despite a missing/forged lockfile — with no test to catch it.

Concrete test cases to add (e.g. by extracting the two run: blocks into scripts/action-lockfile-gate.sh / scripts/action-lockfile-check.sh with *.test.sh siblings, matching the existing release-script pattern):

  • Enforce gate precedence: SOFT_FAIL=true EXIT_CODE=0 LOCKFILE_STATUS=missing → expect exit 1 (regression: soft-fail masks a missing lockfile → job passes). Same for LOCKFILE_STATUS=failed and =error.
  • Enforce gate pass-through: LOCKFILE_STATUS=passed EXIT_CODE=0 → exit 0; LOCKFILE_STATUS=skipped SOFT_FAIL=true EXIT_CODE=1 → exit 0 (skipped must not fail, and must not override the normal soft-fail path).
  • Verify step skip branches: IN_PROJECT set → status=skipped, exit 0; IN_GITHUB_URL=ghes.example.com → status=skipped; IN_GITHUB_URL=https://github.com and ="" → not skipped (host parsing at action.yml:206 must treat the default github.com and *.ghe.com as supported but a GHES host as skipped).
  • Verify step missing-file branch: no .github/workflows/actions.lock present → status=missing, exit 0 (the step itself must never fail the job; the gate does).
  • Summary rendering: for each of passed/failed/missing/skipped/error the correct line is emitted, and an empty LOCKFILE_STATUS emits nothing (action.yml:111).

Without these, the feature's security-critical branches — especially that soft-fail cannot mask a lockfile failure — are unverified and free to regress.

Control: .github/claude-prompts/test-coverage.md · resolve this conversation once addressed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed by TestActionMissingLockfileOnlyWarns. 68101a9 adds the ordering this finding called out: the test now fails if the lockfile case moves below the exit code case in the Enforce gate, which would let a passing score hide a failed lockfile. The skip, missing and error branches are pinned too. This finding predates the change that made a missing lockfile warn instead of failing, so missing no longer exits 1. Summary line rendering stays untested on purpose: it is display only and can't change the job result. The test reads the shape of action.yml rather than running the run: blocks, which were exercised in the e2e cases listed in the PR description.

id: lockfile
if: ${{ inputs.verify-lockfile == 'true' }}
shell: bash
env:
GH_TOKEN: ${{ inputs.github-token }}
IN_PROJECT: ${{ inputs.project }}
IN_GITHUB_URL: ${{ inputs.github-url }}
LOCK_VERSION: ${{ inputs.actions-lock-version }}
run: |
set -uo pipefail
lockfile=".github/workflows/actions.lock"
status() { echo "status=$1" >> "$GITHUB_OUTPUT"; }

if [ -n "$IN_PROJECT" ]; then
echo "::warning::verify-lockfile skipped: project is set, so there is no checkout to verify. Set verify-lockfile: false to silence this."
status skipped; exit 0
fi
host="${IN_GITHUB_URL#https://}"; host="${host#http://}"; host="${host%%/*}"
case "$host" in
""|github.com|*.ghe.com) ;;
*)
echo "::warning::verify-lockfile skipped: gh-actions-lock does not support GitHub Enterprise Server (${host}). Set verify-lockfile: false to silence this."
status skipped; exit 0
;;
esac
if [ ! -f "$lockfile" ]; then
echo "::warning file=${lockfile}::No workflow lockfile at ${lockfile}. Create it from the repository root with 'gh extension install github/gh-actions-lock' then 'gh actions-lock', and commit it. See https://docs.github.com/en/actions/concepts/security/workflow-dependency-lockfiles. To opt out, set verify-lockfile: false."
status missing; exit 0
fi
if ! command -v gh >/dev/null 2>&1; then
echo "::error::verify-lockfile is true but the gh CLI is not available on this runner. Install gh or set verify-lockfile: false."
status error; exit 0
fi
if ! gh extension install github/gh-actions-lock --pin "v${LOCK_VERSION#v}"; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Bug risks

Medium: gh extension install runs unconditionally every run and is not idempotent, so on persistent (self-hosted) runners the second run fails the whole job even when the lockfile verifies. (Medium)

The new Verify workflow lockfile step installs the extension on every invocation:

if ! gh extension install github/gh-actions-lock --pin "v${LOCK_VERSION#v}"; then
  echo "::error::Could not install gh-actions-lock v${LOCK_VERSION#v}. Check actions-lock-version."
  status error; exit 0
fi

gh extension install is not idempotent: if an extension already providing the actions-lock command is installed, it exits non-zero (there is already an installed extension...). It has no "install or skip if present" mode.

Concrete failure scenario (reachable): a repository runs this action on a self-hosted runner (a supported configuration — this is a published action for arbitrary consumers). The gh extensions directory (~/.local/share/gh/extensions) persists between jobs on such a runner.

  • Run 1: extension not present → installs → gh actions-lock --verify succeeds → status=passed. Fine.
  • Run 2 (same runner): the extension is already installed → gh extension install exits non-zero → the step sets status=error and exit 0.
  • The Enforce gate step then hits error) echo "::error::The workflow lockfile could not be verified..."; exit 1, failing the job — despite the lockfile being present and valid. Because the lockfile error/missing/failed cases are evaluated before soft-fail, soft-fail: true does not mask it either.

This makes the check pass once and then permanently break on any runner that reuses state, which is exactly the "behavior across runs" case: a successful first run and a failing identical second run.

Suggested fix: make the install idempotent, e.g. skip when already present —

if ! gh extension list 2>/dev/null | grep -q 'gh-actions-lock'; then
  if ! gh extension install github/gh-actions-lock --pin "v${LOCK_VERSION#v}"; then
    echo "::error::Could not install gh-actions-lock v${LOCK_VERSION#v}. Check actions-lock-version."
    status error; exit 0
  fi
fi

or install/upgrade with the pinned version and tolerate the already-installed exit code (e.g. gh extension install ... || gh extension upgrade ...). Pinning the version should still be applied via --pin / upgrade so the behavior across versions stays deterministic.

Control: .github/claude-prompts/bug-risks.md · resolve this conversation once addressed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing this one. When the extension already installed comes from the same owner, gh extension install prints "already installed" and returns nil (source), so a second run on a persistent runner doesn't fail. It only errors when a different owner's extension provides actions-lock. That's a real conflict, and status=error is the right result there. This comes from reading the cli/cli source, not from a run on a self-hosted runner.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Bug risks

Medium: gh extension install github/gh-actions-lock is not idempotent, so on a persistent/self-hosted runner the second and later runs fail the install, report status=error, and the Enforce gate fails the job even though the lockfile is valid. (Medium)

The verify step installs the extension on every run:

if ! gh extension install github/gh-actions-lock --pin "v${LOCK_VERSION#v}"; then
  echo "::error::Could not install gh-actions-lock v${LOCK_VERSION#v}. Check actions-lock-version."
  status error; exit 0
fi

gh extension install is not idempotent: when the extension is already installed it exits non-zero (e.g. "there is already an installed extension that provides the "actions-lock" command"). GitHub-hosted runners are ephemeral so this is invisible there, but gh stores extensions under the user's home (~/.local/share/gh/extensions), which persists across jobs on non-ephemeral self-hosted runners — an explicitly supported deployment (verify-lockfile defaults to true, and the action documents self-hosted/GHES/air-gapped setups).

Concrete failure scenario: a self-hosted runner runs this action in job A (installs gh-actions-lock v0.4.1 successfully → status=passed). The same runner is reused for job B: gh extension install ... --pin v0.4.1 now fails because the extension already exists → the step sets status=error → the Enforce gate (action.yml:392) turns error into exit 1 with "The workflow lockfile could not be verified." The job fails even though the committed lockfile verifies fine. The emitted message ("Could not install ... Check actions-lock-version") is also misleading, since the version is correct. The same false failure occurs if the action is invoked twice within a single job.

Suggested fix: make the install idempotent — e.g. gh extension install github/gh-actions-lock --pin "v${LOCK_VERSION#v}" --force (which overwrites/ignores an already-installed copy), or guard with gh extension list | grep -q gh-actions-lock before installing, treating an already-present matching version as success rather than error.

Control: .github/claude-prompts/bug-risks.md · resolve this conversation once addressed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing this one. When the extension already installed comes from the same owner, gh extension install prints "already installed" and returns nil (source), so a second run on a persistent runner doesn't fail. It only errors when a different owner's extension provides actions-lock. That's a real conflict, and status=error is the right result there. This comes from reading the cli/cli source, not from a run on a self-hosted runner.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Bug risks

Medium: gh extension install is run unconditionally on every invocation and exits non-zero when the extension is already installed, so on persistent self-hosted runners every run after the first records status=error and fails the job as a supply-chain failure. (Medium)

The Verify workflow lockfile step runs:

if ! gh extension install github/gh-actions-lock --pin "v${LOCK_VERSION#v}"; then
  echo "::error::Could not install gh-actions-lock v${LOCK_VERSION#v}. Check actions-lock-version."
  status error; exit 0
fi

gh extension install returns a non-zero exit code when the named extension is already installed (it prints something like there is already an installed extension that provides the "actions-lock" command). On GitHub-hosted runners each job gets a fresh VM, so this never recurs. But on a persistent self-hosted runner (an explicitly supported scenario — the sibling verify-attestation input even calls out air-gapped/self-managed setups) the gh extensions directory (~/.local/share/gh/extensions / $XDG_DATA_HOME) survives between jobs.

Concrete failure: a self-hosted runner runs this action twice. First run installs the extension and verifies fine. Second run's gh extension install fails because the extension already exists → the step records status error (action.yml:219). The Enforce gate step then matches:

error)  echo "::error::The workflow lockfile could not be verified..."; exit 1 ;;

(action.yml:392) and fails the job — and because this is treated as a "supply chain failure," soft-fail does not mask it. A repository with a perfectly valid, verifying lockfile thus fails CI on every run after the first, with a misleading "could not be verified" message.

Suggested fix: make the install idempotent, e.g. only install when the command is missing, or tolerate the already-installed case:

if ! gh extension list 2>/dev/null | grep -q 'gh-actions-lock'; then
  if ! gh extension install github/gh-actions-lock --pin "v${LOCK_VERSION#v}"; then
    ...status error...
  fi
fi

or run gh extension upgrade github/gh-actions-lock --pin "v${LOCK_VERSION#v}" when it is already present, so a repeated run on a persistent runner does not spuriously fail the job.

Control: .github/claude-prompts/bug-risks.md · resolve this conversation once addressed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing this one, same answer as the two earlier threads. When the extension already installed comes from the same owner, gh extension install prints "already installed" and returns nil (source), so a second run on a persistent runner doesn't fail. The "there is already an installed extension" error only applies when a different owner's extension provides actions-lock, and status=error is the right result there. The nil return dates from cli/cli#8211 (October 2023), so only a gh older than that would fail here. This comes from reading the cli/cli source, not from a run on a self-hosted runner.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Security

Low: the workflow-lockfile verifier installs and executes the github/gh-actions-lock gh extension pinned to a mutable git tag (v0.4.1) rather than an immutable commit SHA, so a moved tag runs attacker code with GH_TOKEN. (Low)

The new step runs:

gh extension install github/gh-actions-lock --pin "v${LOCK_VERSION#v}"
...
gh actions-lock --verify

--pin here resolves to the git tag v0.4.1 (from actions-lock-version, default 0.4.1). Tags are mutable: whoever controls github/gh-actions-lock can move v0.4.1 to a different commit at any time. Because the extension is then executed in the CI job with GH_TOKEN (inputs.github-token, defaulting to github.token — which this action documents as carrying contents:read, actions:read, and in callers often security-events:write), a re-pointed tag causes arbitrary third-party code to run with that token on every consumer's CI.

Concrete scenario: the v0.4.1 tag of github/gh-actions-lock is force-moved (upstream compromise or tag re-point). Every repo using this action with the default verify-lockfile: true downloads and executes the new commit's code with their GH_TOKEN — the exact mutable-ref supply-chain risk this feature exists to detect.

This is inconsistent with the action's own posture: its other third-party dependencies are SHA-pinned (github/codeql-action@6f2ac81..., actions/upload-artifact@cf430e0...), and the Plumber binary itself is verified via SLSA attestation (gh attestation verify), whereas this extension gets neither a SHA pin nor provenance verification.

Suggested fix: gh extension --pin accepts a commit SHA, not just a tag. Default actions-lock-version to a full commit SHA of the intended github/gh-actions-lock release (or additionally verify the extension's provenance after install), so the executed code is anchored to an immutable ref like the action's other dependencies.

Control: .github/claude-prompts/security.md · resolve this conversation once addressed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing this one. The scenario needs v0.4.1 to move, and it can't: that release is immutable (immutable: true on the release, as on the project's other recent releases). GitHub locks the tag and the assets, and a deleted release's tag name can't be reused. The suggested SHA default would also break the step. gh-actions-lock is a precompiled extension, so gh resolves --pin with fetchReleaseFromTag (source), and a release lookup by commit SHA returns 404. This comes from the cli/cli source and the API, not from a real install with a SHA.

echo "::error::Could not install gh-actions-lock v${LOCK_VERSION#v}. Check actions-lock-version."
status error; exit 0
fi
if gh actions-lock --verify; then
status passed
else
echo "::error file=${lockfile}::Workflow lockfile verification failed. See this step's log, then run 'gh actions-lock' locally to fix the lockfile."
status failed
fi
exit 0

- name: Run Plumber analyze
id: run
shell: bash
Expand Down Expand Up @@ -253,11 +309,20 @@ runs:
IN_OUTPUT: ${{ inputs.output }}
PASSED: ${{ steps.run.outputs.passed }}
EXIT_CODE: ${{ steps.run.outputs.exit_code }}
LOCKFILE_STATUS: ${{ steps.lockfile.outputs.status }}
run: |
set -uo pipefail
{
echo "## Plumber"
echo ""
case "${LOCKFILE_STATUS:-}" in
passed) echo "**Workflow lockfile:** ✅ verified" ;;
failed) echo "**Workflow lockfile:** ❌ verification failed" ;;
missing) echo "**Workflow lockfile:** ⚠️ missing (.github/workflows/actions.lock)" ;;
skipped) echo "**Workflow lockfile:** ⚠️ skipped" ;;
error) echo "**Workflow lockfile:** ❌ could not run the verification" ;;
esac
[ -n "${LOCKFILE_STATUS:-}" ] && echo ""
if [ -n "${PASSED:-}" ]; then
status="❌ failed"; [ "${PASSED:-}" = "true" ] && status="✅ passed"
# A degraded run (exit 3: --fail-warnings + a check that could not be
Expand Down Expand Up @@ -315,8 +380,17 @@ runs:
EXIT_CODE: ${{ steps.run.outputs.exit_code }}
SOFT_FAIL: ${{ inputs.soft-fail }}
IN_THRESHOLD: ${{ inputs.threshold }}
LOCKFILE_STATUS: ${{ steps.lockfile.outputs.status }}
run: |
set -uo pipefail
# The workflow lockfile check comes first and soft-fail does not
# mask it: a lockfile that does not verify, or could not be
# checked, is a supply chain failure, not a score below the gate.
# A missing lockfile only warns, in the Verify workflow lockfile step.
case "${LOCKFILE_STATUS:-}" in
failed) echo "::error::Workflow lockfile verification failed. See the 'Verify workflow lockfile' step log."; exit 1 ;;
error) echo "::error::The workflow lockfile could not be verified. See the 'Verify workflow lockfile' step log."; exit 1 ;;
esac
code="${EXIT_CODE:-2}"
case "$code" in
0)
Expand Down
101 changes: 101 additions & 0 deletions cmd/action_template_test.go

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Test coverage

Medium: TestActionMissingLockfileOnlyWarns never asserts the status value emitted by the missing/skipped branches, so a regression that makes a missing lockfile fail the job would ship undetected. (Medium)

TestActionMissingLockfileOnlyWarns is the only test that exercises the new lockfile logic, and it purports to pin "a repository with no workflow lockfile ... gets a warning and keeps going." But it only checks text fragments: the ::warning string is present, no ::error string, the Enforce gate has no missing) arm, and the gate still has failed)/error) arms. It never asserts which status the missing branch actually emits.

At runtime the missing path depends entirely on action.yml:74 emitting status missing (not failed/error), because the gate (action.yml:127-130) fails the job on exactly failed and error. If that line were changed to status failed (or error), every repository without a lockfile would start failing the job — the precise behavior the test claims to guard — yet all of this test's assertions still pass (the warning is still emitted, no error string is added, the gate still contains failed)/error) and still has no missing) arm). The guard gives false confidence.

The same gap applies to the two skipped branches (action.yml:61 project-set and action.yml:68 GHES): they emit status skipped; exit 0 and must not fail the job, but no test asserts the status they produce either.

Suggested test cases:

  • Assert the missing branch pairs its warning with status missing (e.g. the segment after if [ ! -f "$lockfile" ] contains status missing and does not contain status failed/status error), so swapping the status is caught.
  • Assert the project-set and GHES branches emit status skipped.
  • Positively assert the Enforce gate fires exit 1 only under failed)/error) and that skipped/passed/missing/empty do not reach an exit 1, so a stray skipped) ... exit 1 regression is caught.

Control: .github/claude-prompts/test-coverage.md · resolve this conversation once addressed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dfc18de. The test now checks that the missing branch records status missing, that the project and GHES branches record status skipped, and that the Enforce gate's lockfile case exits 1 only on failed and error. Changing status missing to status failed now fails the test.

Original file line number Diff line number Diff line change
Expand Up @@ -87,3 +87,104 @@ func TestActionTemplateGrantsActionsRead(t *testing.T) {
}
}
}

// TestActionMissingLockfileOnlyWarns pins the split in action.yml between
// a repository with no workflow lockfile, which gets a warning and keeps
// going, and a lockfile that fails verification, which fails the job. The
// logic lives in the composite action's run: blocks, so the test reads
// the shape of the file: the verify step warns and records "missing", the
// skip branches record "skipped" while github.com and *.ghe.com go on to
// verify, a verification that cannot run records "error", and the Enforce
// gate step only exits 1 on "failed" and "error", the statuses that must
// fail the job, ahead of the exit code handling where soft-fail applies.
func TestActionMissingLockfileOnlyWarns(t *testing.T) {
raw, err := os.ReadFile("../action.yml")
if err != nil {
t.Fatalf("reading action.yml: %v", err)
}
action := string(raw)

_, verify, found := strings.Cut(action, "- name: Verify workflow lockfile")
if !found {
t.Fatal("action.yml has no Verify workflow lockfile step")
}
skips, afterSkips, found := strings.Cut(verify, `if [ ! -f "$lockfile" ]; then`)
if !found {
t.Fatal("the verify step has no lockfile presence check")
}
missing, afterMissing, found := strings.Cut(afterSkips, "\n fi\n")
if !found {
t.Fatal("the lockfile presence check has no closing fi")
}
setup, afterSetup, found := strings.Cut(afterMissing, "if gh actions-lock --verify; then")
if !found {
t.Fatal("the verify step does not run gh actions-lock --verify")
}
verdict, _, found := strings.Cut(afterSetup, "\n fi\n")
if !found {
t.Fatal("the gh actions-lock --verify check has no closing fi")
}

if got := strings.Count(skips, "status skipped"); got != 2 {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Test coverage

Medium: the lockfile test asserts the skip branches but never pins that a normal github.com host falls through to actually run verification, so a case-pattern regression that silently disables the check on github.com would still pass the test. (Medium)

The verify step's control flow (action.yml:201-227) decides whether the supply-chain check actually runs: the host case arm ""|github.com|*.ghe.com) ;; is the pass-through that proceeds to gh actions-lock --verify, while the *) arm records status skipped and exits. The new test only asserts strings.Count(skips, "status skipped") == 2 (line 118) — it counts the text of the two skip arms but never asserts which hosts take the pass-through arm.

Concrete regression this lets through: suppose a later edit refactors the host case and drops github.com from the pass-through pattern (e.g. the case becomes ""|*.ghe.com) ;;). Every normal github.com run then falls into the *) arm, records status skipped, and lockfile verification is silently never performed — the feature becomes a no-op on the default platform. The test still passes, because the two status skipped text occurrences are unchanged, so the count is still 2.

The same blind spot covers the verify step's error branches (action.yml:213-219): the test confirms the Enforce gate exits 1 on error, but never asserts that the "gh not available" and "install failed" branches actually record status error. A regression that mislabeled one of those as status skipped would let the job pass on a runner where verification could not run — again undetected.

Suggested test additions (same file-reading style as the existing test):

  • Assert the skips region contains a pass-through arm naming github.com and empty host, and that the GHES skip arm is the *) catch-all rather than github.com, so dropping github.com from the pass-through fails the test.
  • Assert the verify step's gh-missing and install-failure branches each contain status error (not status skipped/status missing), pinning that an unrunnable verification fails the job rather than passing silently.

Control: .github/claude-prompts/test-coverage.md · resolve this conversation once addressed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 68101a9. The test now checks that github.com and *.ghe.com take the pass-through arm, that the gh-missing and install-failure branches record status error and no skip, and that the verify result records status passed and status failed. Dropping github.com from the arm, or changing any of those statuses to status skipped, now fails the test.

t.Errorf("the project and GHES branches must record status skipped, found %d", got)
}
if !strings.Contains(skips, `""|github.com|*.ghe.com) ;;`) {
t.Error("github.com and *.ghe.com must go on to verify the lockfile, not take the GHES skip")
}
if !strings.Contains(missing, `echo "::warning file=${lockfile}::No workflow lockfile`) {
t.Error("the verify step must emit a ::warning when the lockfile is missing")
}
if strings.Contains(missing, "::error") {
t.Error("the verify step must not emit an ::error when the lockfile is missing")
}
if !strings.Contains(missing, "status missing") {
t.Error("the verify step must record status missing when the lockfile is missing")
}
for _, status := range []string{"status failed", "status error"} {
if strings.Contains(missing, status) {
t.Errorf("a missing lockfile must not record %q: the Enforce gate would fail the job", status)
}
}
if got := strings.Count(setup, "status error"); got != 2 {
t.Errorf("the gh-missing and install-failure branches must record status error, found %d", got)
}
for _, status := range []string{"status skipped", "status missing"} {
if strings.Contains(setup, status) {
t.Errorf("a verification that cannot run must not record %q: the job would pass unverified", status)
}
}
for _, status := range []string{"status passed", "status failed"} {
if !strings.Contains(verdict, status) {
t.Errorf("the gh actions-lock --verify check must record %q", status)
}
}

_, gate, found := strings.Cut(action, "- name: Enforce gate")
if !found {
t.Fatal("action.yml has no Enforce gate step")
}
beforeCase, afterCase, found := strings.Cut(gate, `case "${LOCKFILE_STATUS:-}" in`)
if !found {
t.Fatal("the Enforce gate has no lockfile status case")
}
if strings.Contains(beforeCase, `code="${EXIT_CODE`) {
t.Error("the Enforce gate must check the lockfile status before the exit code, or a passing score hides a failed lockfile")
}
arms, _, found := strings.Cut(afterCase, "esac")
if !found {
t.Fatal("the Enforce gate's lockfile status case has no esac")
}
for _, status := range []string{"failed)", "error)"} {
if !strings.Contains(arms, status) {
t.Errorf("the Enforce gate must fail the job on lockfile status %q", status)
}
}
for _, status := range []string{"missing)", "skipped)", "passed)", "*)"} {
if strings.Contains(arms, status) {
t.Errorf("the Enforce gate must not act on lockfile status %q", status)
}
}
if got := strings.Count(arms, "exit 1"); got != 2 {
t.Errorf("the Enforce gate must exit 1 on failed and error only, found %d exits", got)
}
}
3 changes: 3 additions & 0 deletions scripts/release-bump-version.test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ inputs:
default: "v0.4.16"
threshold:
default: "100"
actions-lock-version:
default: "0.4.1"
EOF
cat > "$1/templates/plumber.yml" <<EOF
image:
Expand All @@ -42,6 +44,7 @@ if grep -qF "default: \"getplumber/plumber@sha256:${NEW}\" #v0.4.17" "$d/templat
if grep -q "$OLD" "$d/templates/plumber.yml"; then ko "previous digest is gone"; else ok "previous digest is gone"; fi
if grep -qF 'default: "v0.4.17"' "$d/action.yml" && grep -qF '(e.g. v0.4.17)' "$d/action.yml"; then ok "action.yml version and hint bumped"; else ko "action.yml version and hint bumped"; fi
if grep -qF 'default: "100"' "$d/action.yml"; then ok "other defaults untouched"; else ko "other defaults untouched"; fi
if grep -qF 'default: "0.4.1"' "$d/action.yml"; then ok "actions-lock-version default untouched"; else ko "actions-lock-version default untouched"; fi
if [ "$(cat "$d/README.md")" = " - uses: getplumber/plumber@<version>" ]; then ok "README untouched"; else ko "README untouched"; fi
no_backup "$d" "no backup files left"

Expand Down
Loading