Skip to content

fix(ci): run the selected suites for uncommitted changes as for committed ones - #10681

Merged
kyle-sexton merged 3 commits into
mainfrom
fix/check-affected-uncommitted-selection
Oct 28, 2026
Merged

kyle-sexton merged 3 commits into
mainfrom
fix/check-affected-uncommitted-selection

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

No related issue: a peer found it while checking #10675 before its push; no issue was filed.

Summary

scripts/check-affected.sh --uncommitted --tests never ran the selected suites. It re-runs itself on a scratch snapshot of the work tree with CHECK_DATA_ROOT naming that snapshot, so the inner run treats the snapshot as a foreign tree and skips tests with SKIP tests (runs the checked tree's own test code). That skip is the pre-push hook's guard for a merge-base copy. Here it hid two failing digest pins in plugins/planning/tests/interview-defenses.test.sh over an uncommitted plugins/planning/skills/interview/SKILL.md edit. Test selection was fine: scripts/affected-tests.sh already reads the work tree and picks that suite.

Fix

  • The outer --uncommitted run passes CHECK_AFFECTED_SNAPSHOT_OF=<its own tooling tree>, but only when it is checking its own tree. The inner run clears the tests skip only when that value equals its own tooling tree.
  • In that case the suites run from the tooling tree, through the same affected() step a committed run uses. affected-tests.sh reads the work tree's uncommitted and untracked edits there, the same content as the snapshot, and the scratch snapshot has no node_modules.
  • setup-env.sh --node now gets --root "$tools", so it never installs into the scratch snapshot.
  • A copy checking another tree (the pre-push hook) still skips tests. A new case covers it.

Verification

  • bash scripts/check-affected.test.sh: PASS=79 FAIL=0. The new case --uncommitted --tests runs the suites over the uncommitted change failed before the fix with SKIP tests (runs the checked tree's own test code) (PASS=78 FAIL=1).
  • End to end, with a temporary edit inside the interview skill's digested Stance section: before the fix, --uncommitted --tests printed SKIP tests (runs the checked tree's own test code). After the fix it printed FAIL tests (exit 1), from the Stance section's digest pin in interview-defenses.test.sh.
  • bash scripts/check-affected.sh --tests on the commit: 11 passed, 5 skipped. Tests is a SKIP only because 2 selected Node suites are left to their CI lane.
  • Left out: validate-plugins, because no plugin is touched. Plain --uncommitted without --tests still runs no suite, as before.

Related

N/A

🤖 Generated with Claude Code

kyle-sexton and others added 3 commits October 28, 2026 11:33
…tted ones

`check-affected.sh --uncommitted --tests` re-runs itself on a scratch
snapshot of the work tree, so the inner run saw a data tree other than
its own and skipped tests as foreign code. A digest pin over an
uncommitted skill edit then went unchecked.

The snapshot is the tooling tree's own work tree, so the outer run now
marks it, and the inner run runs the suites from the tooling tree,
whose selector reads the same uncommitted edits. A copy checking
another tree still skips them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 28, 2026 16:03
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T16:06:20.022256Z 436ca97 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed 436ca9740a493e909f9ee6318f833a9a21c25d07: 0 finding(s), 0 as inline comments.

No findings. I read the full diff (scripts/check-affected.sh, scripts/check-affected.test.sh), the PR context, and the surrounding code in check-affected.sh, affected-tests.sh and setup-env.sh. I did not run any tests; the PASS=79 FAIL=0 and end-to-end results in the PR body are author-claimed and unverified by me.

What I checked:

  • Which tree the inner run is in: check-affected.sh:139 does cd "$top", so the inner snapshot run's pwd -P is the scratch tree. That is why it was foreign=1 and skipped tests, and why the snapshot flag now overrides that skip.
  • Which tree the tests read: affected-tests.sh:225 does cd "$SCRIPT_DIR/..", which is the tooling tree. So the selector reads the real work tree's uncommitted and untracked edits, as the PR describes, and not the scratch snapshot.
  • Pre-push guard: the outer run sets CHECK_AFFECTED_SNAPSHOT_OF only when its pwd -P equals $tools (line 201). A merge-base copy checking another tree therefore passes an empty value and still skips tests. The new second test case covers that.
  • Environment variable handling: the inner run reads the variable at line 208 and unsets it at line 209. The non---uncommitted path never sets it, so it does not leak into suites. setup-env.sh supports --root (lines 30–31), and --root "$tools" matches its previous default for a non-foreign run.
  • Residual risk (not a finding): an externally exported CHECK_AFFECTED_SNAPSHOT_OF=<tools> in a foreign run would bypass the skip. That needs a deliberate override of a script-internal variable and is unlikely in practice.

No closing issue (the PR states none). No manifests changed. No cost or critical-security hold applies.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-security-review: 1 inline finding(s) on 436ca9740a493e909f9ee6318f833a9a21c25d07; the summary is in this lane's pull request comment.

claude-lane: claude-security-review

Comment thread scripts/check-affected.sh Resolved
@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 436ca9740a493e909f9ee6318f833a9a21c25d07: 1 finding(s), 1 as inline comments.

Security review of #10681 (scripts/check-affected.sh and its test). Neither file matches a trusted path, and the change touches no trust control, manifest or instruction surface.

What the change does. The outer --uncommitted run exports CHECK_AFFECTED_SNAPSHOT_OF=<tooling tree>, but only when its cwd is that tooling tree (scripts/check-affected.sh:200-203). The inner run clears the foreign && !snapshot tests skip only when that value equals its own $tools (:207-209, :582). It then unsets the variable. The suites run from $tools with CHECK_DATA_ROOT unset (:616-618). setup-env.sh --node is now pinned to --root "$tools", so npm ci never runs in the scratch snapshot.

Checked and not a finding.

  • The merge-base-copy case, where cwd is the branch tree and $tools is the copy. The outer run sets own="", so the skip stays. The new test covers this. An attacker would have to set the environment variable themselves, and anyone who can do that already has code execution.
  • The snapshot path. Test code runs from the user's own work tree, which holds the user's own edits. The snapshot adds no foreign code.

critical-findings: 0

Finding (SUGGESTION): a standing guard now rests on a premise that no longer holds for one invocation. docs/conventions/pr-pipeline/pr-pipeline.schema.json:523-526 (no-head-test-check) blocks checks that run the checkout's test files in the write runner's job, which holds the lane token and id-token: write. It allows scripts/check-affected.sh --tests on the stated premise that the script "skips the branch's tests". That is still true for a base copy. It is no longer true for check-affected.sh --uncommitted --tests run from the checkout's own tree, which now runs the checked-out test suites. The schema pattern only blocks the other test scripts. No current config uses this form: docs/conventions/pr-pipeline.yaml:294 lists only --uncommitted, and .github/actions/resolve-config/self-checks.test.mjs:80 pins --tests as a base-copy check. The path would need a later change to pr-pipeline.yaml, which is a hold path, so I rate it SUGGESTION. Consider adding check-affected.sh .*--uncommitted.*--tests to the schema's blocked pattern. Also update the schema description, since --uncommitted --tests no longer skips the branch's tests.

Spend: none that bills. The change only makes local and lane pre-push self-checks run more test suites.

@kyle-sexton
kyle-sexton enabled auto-merge October 28, 2026 17:17
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit 9d2beb6 Oct 28, 2026
73 checks passed
@kyle-sexton
kyle-sexton deleted the fix/check-affected-uncommitted-selection branch October 28, 2026 18:00
@github-actions

Copy link
Copy Markdown
Contributor

Post-merge mutation testing

Killed 1, survived 0, timeout 0, no coverage 0, invalid 0, not run 7.
Covered-code score 100%; mutation score 100%.

File Killed Survived Timeout No coverage Invalid Not run
scripts/check-affected.sh 1 0 0 0 0 7

The run's summary has the full report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant