[kyle-sexton]

Refs: #7148

Summary

The setup-writes-only-confirmed-change eval case only checked that the team file exists. A reply that wrote the default convention instead of the one the user confirmed got full marks. A host run of the #7148 paid evals showed the case had no with/without difference until this grader was added.

Fix

• New regex grader team-file-holds-confirmed-pattern reads docs/conventions/source-control.yaml and requires the confirmed ticket-prefixed subject_pattern.
• Each of the case's four graders now has pass and fail samples, which validate-cases.py checks offline.
• Eval-only change: no skill, hook or script changes.

Verification

• claude plugin eval on the source-control plugin (Claude Code 2.1.314), --tag babysit-prs babysit-loop setup --runs 3, both arms, subscription login: setup scored 1.00 with the plugin and 0.75 without, in all 3 runs of each arm.
• validate-cases.py plugins/source-control/evals: no FAIL. Swapping the new grader's pass and fail samples makes it FAIL on every sample, so the samples are checked.
• scripts/check-affected.sh --tests on the committed head: 12 passed, 4 skipped.
• Left out: the run-validity.py verdict for the paid run is INVALID, for reasons outside this change (fix(evals): run-validity's session-started check fails runs whose Bash calls exit non-zero with output #11290, and a with-arm permission denial in the babysit-prs case). Details are in the test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148 host-run record.

Related

• test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148 stays open. The babysit cases still reach the real gh in the no-plugin arm and in babysit-loop: test(source-control): babysit eval cases reach the real gh in the no-plugin arm and in babysit-loop #11289.
• run-validity session-started false positive: fix(evals): run-validity's session-started check fails runs whose Bash calls exit non-zero with output #11290.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T15:41:37.036866Z	93cf743	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-review reviewed 93cf743ed6557c49b54822d4c7d754519205da9c: 1 finding(s), 0 as inline comments.

Eval-only change. It adds a regex grader that checks the team file holds the confirmed ticket-prefixed subject_pattern, plus pass/fail samples for the case's four graders. I read the whole diff and traced the regex against each new sample by hand. I did not run validate-cases.py or any eval; the PR's verification numbers are author-claimed and unverified. No closing issues are listed (Refs: #7148). The PR touches no trusted paths, so no hold. bump: none matches the PR's description of the change, and the .changes fragment format matches other fragments in .changes/. Manifest items: none hit.

Regex tracing found one minor issue: a sample marked "pass" is invalid YAML (inline comment below). The change otherwise looks sound. The unclosed-quote, wrong-key, dropped-ticket-number and empty-file fail samples all fail the regex as intended.

critical-findings: 1

Findings not posted inline (their lines are not in the diff):

• plugins/source-control/evals/setup-writes-only-confirmed-change/samples/team-file-holds-confirmed-pattern.json:67 (RIGHT): This pass sample is not valid YAML. In a plain (unquoted) scalar, : after \d+ makes a YAML parser raise "mapping values are not allowed here". The pattern contains : , so it can only be written quoted. The grader's (["']?) group accepts the unquoted form, so a reply that writes an unparseable team file would score as having the confirmed pattern. The fail sample on line 73 is excluded for "would not parse", but this one fails to parse for the same reason. Suggested fix: make the quote group mandatory ((["'])) and change this sample to a quoted form with a trailing comment. Alternatively, move it to fail and say why.

[github-actions]

claude-security-review reviewed 93cf743ed6557c49b54822d4c7d754519205da9c: 0 finding(s), 0 as inline comments.

No security issues found in this pull request.

The change adds one eval grader (team-file-holds-confirmed-pattern.md), four sample JSON files and a changelog fragment. All of it sits under plugins/source-control/evals/ and .changes/. None of these files match a trusted path, and none touch a workflow, action, token, hold or manifest.

• Grader regex: It has no nested or overlapping quantifiers, so catastrophic backtracking is not a concern. It only reads docs/conventions/source-control.yaml from the eval's own sandbox.
• Samples: They are static data. They contain no commands and no secrets.
• PR context: The description has no embedded instructions.
• Plugin manifests: No plugin.json or marketplace.json changed, so there are no manifest <field> lines.
• Instruction surfaces: No standing instruction was deleted or softened.
• Spend: The description reports a paid host run of claude plugin eval on a subscription login. The change itself only adds a grader and samples, and any later eval run draws on subscription usage. It is not a hold.

critical-findings: 0