[kyle-sexton]

Refs: #12043

Summary

Two evals cases carry a skill-fired grader (tool_used, tool Skill, expects evals:plugin-eval) yet fired it in 0 of 3 runs on main with skill loaded: yes, so their with-arm deltas measured nothing:

• sandbox-wsl2-missing-socat: run 41841302783
• target-before-json: run 41841355473

Fix

• Each prompt now opens "With the evals:plugin-eval skill: ...", as test(code-tidying): name audit-dead-code in adc-read-only-delete-request #13204 did. Both cases test answer content (the socat preflight verdict; target before --allow-tools), not triggering, so naming the skill keeps their purpose. Each description now says the case tests behavior once the skill is loaded, not triggering.
• The grader already expects the right skill (plugins/evals/skills/plugin-eval), so it is unchanged. The socat-missing and target-first graders are unchanged.
• One bump: none changelog fragment.

Triggering finding, reported on #12043 and not fixed here: plain questions about a WSL2 sandbox missing socat, and about claude plugin eval argument order, do not trigger /evals:plugin-eval.

Verification

• validate-cases.py plugins/evals: 0 FAIL (WARNs only, judge-calibration notices these cases already had).
• scripts/check-affected.sh --tests on the committed head: 13 passed, 4 skipped over 3 changed files.
• Local run of each case at the PR head (claude plugin eval <archive of plugins/evals> --case <name> --runs 1 --allow-tools Read Glob Grep Skill, Claude Code 2.1.318, with-without ablation):
  • sandbox-wsl2-missing-socat: with-arm skill-fired passed ("Skill called 1x", input matched evals:plugin-eval), socat-missing passed; without-arm failed it. Delta 1.0.
  • target-before-json: with-arm skill-fired passed ("Skill called 1x"), target-first passed; without-arm failed it. Delta 1.0.
• One run per case is a firing check, not a measured gain; the post-merge 3-run dispatch is the measurement.

Remaining on #12043

• After merge, re-dispatch evals sandbox-wsl2-missing-socat and target-before-json, read the health line, and link the runs on feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405.

Related

• feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405 (survey list), test(code-tidying): name audit-dead-code in adc-read-only-delete-request #13204 (reference repair).

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-04T13:56:59.284648Z	e2e5dfd	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed e2e5dfddf59dcc1c2f672aef344d19fd03a05857: read 3 of 3 changed file(s); 0 finding(s), 0 as inline comments.

Security review: no security issues found

I read every hunk of all 3 changed files. None of them match a trusted path.

• .changes/evals/test-12043-name-plugin-eval-in-two-cases-4c81d2e7.md: a new bump: none changelog fragment. It has no executable content.
• plugins/evals/evals/sandbox-wsl2-missing-socat/prompt.md and plugins/evals/evals/target-before-json/prompt.md: each eval case's description gains one sentence, and each prompt body now starts with "With the evals:plugin-eval skill:". The front-matter fields that affect a run (tags, runs, max_turns, expected_outcome) are unchanged, and no allowed_tools grant changes.

There are no security-relevant changes:

• No trust control, token scope, kill switch, merge hold or plugin manifest is touched.
• No standing instruction is deleted or weakened.
• No code path is added that untrusted input could reach.

The PR description contains no embedded instructions aimed at this lane. This is the first review of this pull request, so the full review ran rather than blocking-only. The skip gate's "no security-relevant surface" condition applies.

Spend: re-dispatching these two eval cases after merge uses prepaid API or subscription usage only. That is not a hold.

critical-findings: 0

[github-actions]

claude-review reviewed e2e5dfddf59dcc1c2f672aef344d19fd03a05857: read 3 of 3 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #13245 (complete, 3 of 3 files read)

Scope: This change edits eval prompts only. The sandbox-wsl2-missing-socat and target-before-json prompts now start with "With the evals:plugin-eval skill:", each description gains "It tests behavior once the skill is loaded, not triggering.", and there is one bump: none changelog fragment.

Checks made:

• The skill-fired grader in target-before-json/graders/skill-fired.md expects Skill with (evals:)?plugin-eval, which matches the skill the new prompt names. The PR leaves the graders unchanged, as it says.
• The new description sentence uses the same wording as the earlier repairs in plugins/developer-experience/evals/s7-act-after-yes, s4-migrator-build and plugins/code-tidying/evals/adc-read-only-delete-request. That keeps the change in line with the existing pattern (test(code-tidying): name audit-dead-code in adc-read-only-delete-request #13204).
• The answer-content graders and samples (socat-missing, target-first) are untouched. The new prefix does not change what those graders check.
• Path-scoped rule .claude/rules/eval-case-transcripts.md: the prompts are synthetic. They hold no raw transcript and no identifying detail.
• bump: none is the right level, since nothing a plugin user runs changes. No plugin manifest changed.
• Closing issue: the PR only references feat(evals): repair or retire eval cases whose with-arm never loaded the skill #12043 ("Refs") and closes no issue, so there were no acceptance criteria to check. The PR body says triggering on plain questions is still not fixed and is tracked on feat(evals): repair or retire eval cases whose with-arm never loaded the skill #12043.

Verification: The PR body's results for validate-cases.py, check-affected.sh and the local runs are the author's claims. I did not verify them.

Spend: After merge, re-dispatching these two eval cases uses prepaid API credits (3 runs each). That is not a hold.

No findings.