Skip to content

test(source-control): name the pull-request skill in the dirty-worktree eval case - #12915

Merged
kyle-sexton merged 1 commit into
mainfrom
host-run/12655-pr-eval-cases
Nov 3, 2026
Merged

kyle-sexton merged 1 commit into
mainfrom
host-run/12655-pr-eval-cases

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Refs: #12655

Summary

The pull-request-dirty-worktree-stashes-before-checkout eval case now names the source-control:pull-request skill in its prompt. The case now tests the stash-before-checkout behavior, not triggering. The #12655 paired eval of #12739 needs every case's with-arm run to load the skill, and with its plain prompt this case never did.

Triggering finding, reported separately and not fixed here: with the plain prompt on main (1e321c6), the case fired the skill in 0 of 4 with-arm runs (Claude Code 2.1.317, claude-opus-5-5). In each run the model ran git stash push --include-untracked and then git switch main itself, without calling the Skill tool. The skill's description and trigger probes need this (/skill-quality:check measure-invocation).

Fix

  • One prompt line: "PR 482 is merged. With the source-control:pull-request skill, reuse this worktree ...". perf(source-control): move the commit skill's rationale out of its hub #12704 made the same change to three commit cases.
  • A bump: none changelog fragment.
  • pull-request-ready-hands-pr-to-explain-change keeps its plain prompt. It already loads the pull-request skill. Its missed call is the graded hand-off to explain-change, so naming that skill would give the model the answer.

Verification

Remaining on #12655

Related

🤖 Generated with Claude Code

…ee eval case

On main the case's with-arm run never called the Skill tool: the model
stashed and switched branches with plain git. Naming the skill makes the
case test the stash-before-checkout behavior, not triggering, so the
#12655 paired eval can score it.

Refs: #12655

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T19:08:12.499674Z f400737 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

claude-security-review reviewed f400737f8a9c475bd48567187ac6b1536d2da54d: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #12915

I found no security issues.

I read both changed files in full:

  • .changes/source-control/test-12655-dirty-worktree-names-skill-5b2e9c41.md is a new changelog fragment with bump: none. It is prose only.
  • plugins/source-control/evals/pull-request-dirty-worktree-stashes-before-checkout/prompt.md changes one line of an eval prompt so it names the source-control:pull-request skill. The frontmatter and expected_outcome stay the same.

Neither file matches a trusted-path glob, and the PR changes no trust control, plugin manifest, workflow, token scope or standing instruction. The diff adds one eval prompt line and removes nothing that guards behavior, so the instruction-surface lens finds nothing to flag. The PR context contains no embedded directives aimed at this lane.

No prior findings apply: 0 rounds so far, cap 3, not past the cap.

critical-findings: 0

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

claude-review reviewed f400737f8a9c475bd48567187ac6b1536d2da54d: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Code review: no findings

I read the whole change in both files in the diff:

  • plugins/source-control/evals/pull-request-dirty-worktree-stashes-before-checkout/prompt.md: one prompt line now names the source-control:pull-request skill. Three commit eval cases already name the commit skill in their prompts, and this line matches that style (commit-never-stage-everything, commit-hook-rejection-no-bypass and commit-trailer-model-from-session). The frontmatter is unchanged, and expected_outcome still grades stash-before-checkout, which is the behavior the PR says the case now tests.
  • .changes/source-control/test-12655-dirty-worktree-names-skill-5b2e9c41.md: a bump: none fragment with a non-empty reason. scripts/check-changelog-fragments.sh and scripts/release-plugins.sh both accept this form. A change to an eval prompt alone ships nothing to plugin users, so none is the right level.

Path-scoped rules: .claude/rules/eval-case-transcripts.md applies to the eval file. The prompt is a synthetic case and contains no transcript or identifying detail, so it meets the rule.

Closing issue: the PR only references #12655 and closes no issue, so there are no acceptance criteria to check. The PR context file contains no embedded instructions aimed at this lane.

Verification: I ran no evals or tests. The firing check, the paired-round verdicts and the check-affected results in the PR description are the author's claims and are unverified here.

Spend: none in this change. Running the eval case uses only prepaid model credits.

Prior findings: none. This is round 1 of 3, so the cap did not apply.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant