[kyle-sexton]

Refs: #9966

Summary

The four /harness-ops:plugins-audit eval cases are now claude plugin eval cases under plugins/harness-ops/evals/, and three of them carry a fleet fixture. These are the files the #8206 output arm ran on (result on #9966). They were in a scratch copy only; this PR keeps them, as the hub asked (H48 (b)).

Fix

• Converted skills/plugins-audit/evals/evals.json with convert-evals-json.py and removed the legacy file. The four case directories match a fresh conversion of main's file, apart from the run settings below.
• audit-issues-zero-mutating-calls, divergence-report-collapses-bulk-names-conflicts and model-starts-audit-not-sync each get a scaffold.sh and context.scaffold_script. The scaffold writes the run's home with known_marketplaces.json, a catalog, installed_plugins.json, settings and a harness-ops install record, so the default marketplace resolves. The divergence case gets the fleet its prompt names: 29 plugins whose scopes differ, 2 same-version multi-scope, and 1 settings-only delisted id.
• All four cases list Bash in allowed_tools. The three scaffolded cases say in their description that they need --scaffold --allow-tools Bash with the real node first on PATH.

Verification

• validate-cases.py plugins/harness-ops/evals exits 0 with no FAIL. Its warnings are the converter's: judge-only graders, and samples that need judge calibration.
• diff -r of a fresh convert-evals-json.py run against these directories shows only the scaffold.sh files, the scaffold_script lines and Bash in allowed_tools.
• Run record, Sonnet 5.5, judge Opus, 3 runs per arm: with the plugin, three of the four cases scored at least 0.8; the noise report gave +0.45, interval +0.26 to +0.65.
• Left out: a re-run on this exact tree. The case files are the ones measured, with only a description line added to three prompts' frontmatter.

Remaining

• test(developer-experience): port eval cases S3 and S6 #9966 keeps batch A's case builds, batch B, fix(guardrails): decide whether a literal '/' in allowed roots counts as a root under Git Bash #8414 and the T3 graders.
• rendered-report-is-pasted-not-restated stayed under its 0.8 bar (0.6, 0.8, 0.4). Its prompt narrates a render with other numbers, so it has no fixture yet.
• test(developer-experience): port eval cases S3 and S6 #9966 closes when its listed ports and builds have landed.

Related

• feat(harness-ops): split plugins audit into its own model-invocable skill #8206: the plugins-audit split whose output arm these cases measured.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-05T05:43:20.556010Z	7acfb23	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed 13b4af188b031b551ec58f9f6ad5d427819144d0: read 30 of 30 changed file(s); 0 finding(s), 0 as inline comments.

Security review: no security issues found

This pull request moves the four /harness-ops:plugins-audit eval cases from the skill's old evals.json into native claude plugin eval case directories. It also adds a fixture scaffold.sh to three of them. Every changed file is eval data, a test fixture or a change note. The diff changes no workflow, trust control, merge hold, manifest, hook or standing instruction.

What I checked

• The three scaffold.sh files. They write the fixture fleet under $HOME. Each one first resolves the account's real home with getent passwd or dscl and refuses to run when HOME points there. It refuses when it can't look the real home up, and refuses to overwrite any file it would write (the refuse checks on lines 13–23 and 31/33 of each script). The divergence scaffold uses one base directory, $PWD, throughout and adds $HOME/other-project to its no-overwrite check.
  • Every interpolated value is a fixed fixture name or a path built from $HOME, $PWD or the script's own directory. No pull request, issue or comment text reaches them, so I found no injection path.
  • The harness-ops install record's installPath points at the checkout's own plugins/harness-ops directory. That is the code under test, not an outside source.
• allowed_tools now includes Bash in all four prompt.md files. This only affects eval runs of fixed prompts that live in the repository. It gives untrusted text no new way into a credentialed lane and grows no token scope.
• Deleted skills/plugins-audit/evals/evals.json. It holds eval definitions, not standing instructions. The same cases are recreated under evals/, so no guardrail is removed.
• Manifests. The diff changes no plugin.json or marketplace.json. The .changes entry is bump: none.
• Trusted paths. No changed file matches a TRUSTED PATHS glob.

Settled thread. My earlier finding on plugins-audit-audit-issues-zero-mutating-calls/scaffold.sh (the fixture home was guessed with no guard against the real home) was resolved in 13b4af1, marked lines changed since resolution: no. I confirmed the head code matches the owner's description of the fix, in all three scaffolds. I am not raising it again.

Identical files. I read the first file of each identical-copy group and skipped the rest. None of the copies sits at a path that would change what its content does. plugins-audit-model-starts-audit-not-sync/scaffold.sh has the same blob as the zero-mutating-calls scaffold but is not on the identical list, so I read it separately.

Spend. Adding Bash to the eval cases and adding scaffolds may lengthen eval runs a little. That uses only prepaid API or subscription usage, and there is no billable change.

critical-findings: 0

[github-actions]

claude-review: 1 inline finding(s) on 7acfb2368e8903438a6d575f1abf8500c11b847b; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed 13b4af188b031b551ec58f9f6ad5d427819144d0: read 3 of 3 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #13442, incremental (7acfb23 → 13b4af1)

No new findings in this push. This review was blocking only, because this lane already reviewed an earlier head. I read every hunk of all 3 files in scope, which are the three plugins-audit-* scaffold.sh files.

The earlier finding is fixed. That finding said the scaffolds guessed the fixture home as $(cd .. && pwd), had no guard against the real home and overwrote files without checking. The owner resolved it, and this push carries the fix. Each scaffold now:

• writes only under $HOME.
• refuses when HOME is unset or not a directory.
• refuses when HOME resolves (pwd -P) to the account's real home from getent passwd or dscl, and refuses when neither lookup tool exists.
• refuses when any target already exists, before writing anything.

The targets checked are the two plugin-state JSON files, the user and workspace settings*.json files, the marketplace directory, and in the divergence scaffold also other-project.

In the divergence scaffold, EVAL_FIXTURE_CWD is gone and the script uses $PWD throughout, so the two-directory split is fixed as well. A grep of the three scaffolds finds no remaining use of the old home or cwd variables, which would have aborted under set -u.

Edge cases I checked:

• If getent fails, pipefail plus set -e stop the script before any write.
• If the real-home directory does not exist, its cd yields an empty string, the comparison passes, and that is correct because HOME cannot be that directory.
• The final printf '{}\n' >"$f" loop now overwrites without the old -s check. That is safe, because the existence guard has already refused any of those paths that existed.

Not checked by me. The owner says they ran these by hand: a fresh temporary home passes, a rerun refuses, HOME=<real home> refuses, and the fixtures are byte-identical after normalising paths. They also say the runner's HOME has no .claude yet, so the runner itself trips no refusal. These are author claims, and I did not run anything.

Spend (not a finding, not a hold). Running these eval cases uses prepaid API credit only. This push does not change that.

Other checks. There is no closing issue, so there are no acceptance criteria to check. No plugin manifest changed.