Repository navigation
test(source-control): name the pull-request skill in the dirty-worktree eval case - #12915
Conversation
…ee eval case On main the case's with-arm run never called the Skill tool: the model stashed and switched branches with plain git. Naming the skill makes the case test the stash-before-checkout behavior, not triggering, so the #12655 paired eval can score it. Refs: #12655 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
claude-security-review reviewed Security review: PR #12915I found no security issues. I read both changed files in full:
Neither file matches a trusted-path glob, and the PR changes no trust control, plugin manifest, workflow, token scope or standing instruction. The diff adds one eval prompt line and removes nothing that guards behavior, so the instruction-surface lens finds nothing to flag. The PR context contains no embedded directives aimed at this lane. No prior findings apply: 0 rounds so far, cap 3, not past the cap. critical-findings: 0 |
|
claude-review reviewed Code review: no findingsI read the whole change in both files in the diff:
Path-scoped rules: Closing issue: the PR only references #12655 and closes no issue, so there are no acceptance criteria to check. The PR context file contains no embedded instructions aimed at this lane. Verification: I ran no evals or tests. The firing check, the paired-round verdicts and the Spend: none in this change. Running the eval case uses only prepaid model credits. Prior findings: none. This is round 1 of 3, so the cap did not apply. |
Refs: #12655
Summary
The
pull-request-dirty-worktree-stashes-before-checkouteval case now names thesource-control:pull-requestskill in its prompt. The case now tests the stash-before-checkout behavior, not triggering. The #12655 paired eval of #12739 needs every case's with-arm run to load the skill, and with its plain prompt this case never did.Triggering finding, reported separately and not fixed here: with the plain prompt on main (1e321c6), the case fired the skill in 0 of 4 with-arm runs (Claude Code 2.1.317,
claude-opus-5-5). In each run the model rangit stash push --include-untrackedand thengit switch mainitself, without calling the Skill tool. The skill's description and trigger probes need this (/skill-quality:check measure-invocation).Fix
bump: nonechangelog fragment.pull-request-ready-hands-pr-to-explain-changekeeps its plain prompt. It already loads the pull-request skill. Its missed call is the graded hand-off toexplain-change, so naming that skill would give the model the answer.Verification
--runs 1 --ablation none --scaffoldon the before arm over all 21pull-request-*cases: 11 of 11 non-slash cases loaded the pull-request skill, this case included. The 10ci-modecases open with the slash command. The ready case never calledexplain-change.--runs 2 -j 2 --scaffold --ablation none --model claude-opus-5-5, on both arms with this case fix: run-validityverdict: VALIDon both arms. noise-reportcompare verdict: non-inferior at margin 0.05. The record goes on perf(source-control): cut the pull-request skill body through progressive disclosure #12655.scripts/check-affected.sh --testson the committed head: 12 passed, 4 skipped over the 2 changed files.Remaining on #12655
pull-request-ready-hands-pr-to-explain-change, the ready flow calledexplain-changein 0 of 2 runs per arm, on both before and after.reference/ready-for-review.md2.5.5 gives the instruction in both arms.Related
🤖 Generated with Claude Code