[kyle-sexton]

Closes #12880

Summary

#12880 reported that an eval run cannot read the loaded plugin copy's own files. Host trials on Claude Code 2.1.317 do not reproduce that: a run reads, globs and greps its own plugin copy by absolute path, with and without a Bash grant. Only a pathless or workspace-relative lookup finds nothing, because it searches the run's empty workspace. /evals:plugin-eval's case-authoring guidance said a reference read "can be denied"; this corrects it.

Fix

• plugins/evals/skills/plugin-eval/reference/case-authoring.md: the reference-spoke checklist item names the measured layout (a plugin copy staged outside git, under the worktree root or on a /run/user/<uid> tmpfs, read by absolute path with or without a Bash grant), says other copy locations are unmeasured, and says a pathless lookup finds nothing; it cites fix(evals): eval runs cannot read the plugin copy's own files, so skills that use them measure low #12880 with the Claude Code version, an as-of date and a recheck trigger on the plugin-evals isolation section.
• plugins/evals/skills/plugin-eval/scripts/test_run_validity.py plus fixture traces/self-lookup-allowed.jsonl: a synthetic trace where Read, Glob and Grep of the plugin copy succeed with no denial; the test asserts check permission denials: PASS and verdict VALID without --own-copy-search-warns. It passed against unchanged run-validity.py, so the script needs no change; the existing denied-lookup cases still pass.
• An evals patch changelog fragment.

Verification

Trials (results on #12880: part 1, part 2), each judged VALID by run-validity.py with no permission denial:

Leg	Plugin copy	Bash	Read / Glob / Grep by absolute path	Relative ../../ link	Pathless Glob
1	worktree root	no	3 of 3	not run	not run
2	worktree root	yes	3 of 3	1 of 1	"No files found"
3	/run/user tmpfs	yes	3 of 3	1 of 1	"No files found"

Bash legs ran through plugins/evals/skills/plugin-eval/scripts/docker-links.py (the #8378 swap); find ~/.docker -mindepth 1 -type l listed the same 2 links before and after.

python3 plugins/evals/skills/plugin-eval/scripts/test_run_validity.py: 151 tests OK. scripts/run-ruff.sh check on the test file: clean. scripts/check-affected.sh: 14 passed, 1 skipped.

Related

• planning: plan_stress_test per mode, settled by the devils-advocate eval #12478 (where the denials were first seen; its traces are gone, so their exact inputs cannot be re-read)
• feat(evals): run plugin-eval suites on Windows, WSL, cloud sessions and GitHub Actions #8378

🤖 Generated with Claude Code

https://claude.ai/code/session_015te8CoibZv2y2Zsidz4jvz

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T23:02:54.684845Z	ddf2903	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed 3df77e761775ebdcf3524f2adfe64f99a06919d2: read 4 of 4 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #12976 (incremental, ddf2903..3df77e7)

No security issues found.

I read every hunk of all 4 files in the incremental scope:

• .changes/evals/fix-12880-eval-own-file-guidance-719afae4.md: a prose-only edit to the changelog fragment. It now names the measured layout, a plugin copy staged outside git, and says other copy locations are unmeasured.
• plugins/evals/skills/plugin-eval/reference/case-authoring.md: the same point in the checklist item. It now says absolute-path Read/Glob/Grep was shown to work only for a copy staged outside git, under the worktree root or on a /run/user/<uid> tmpfs. It links to the skill's Worktree isolation section and says other locations are unmeasured.
• plugins/evals/skills/plugin-eval/scripts/fixtures/run-validity/traces/self-lookup-allowed.jsonl (new): a synthetic trace with made-up /tmp/eval-ctl/... paths, no secrets and no denials.
• plugins/evals/skills/plugin-eval/scripts/test_run_validity.py: one new unit test. It asserts that an allowed lookup of the plugin's own copy passes the denial check and comes out VALID without the flag. It also updates the fixture docstring.

Instruction-surface check. The case-authoring change narrows a factual claim. Before, it said a run could read its own copy in general; now it says this was measured only for the named layout. It removes no rule that prevented an action. The rule on the line above still says the eval directory stays hidden and not to defeat that with add_dirs. "Grade on terms the hub itself carries" is also unchanged. No protected class applies, so I did not consult the exception register.

Trusted paths and controls. None of the 4 files matches a trusted-path glob. The diff touches no trust control, merge hold, token scope, kill switch, workflow or plugin manifest. The test and fixture change no runtime code: the test drives run-validity.py, which this PR leaves unchanged.

PR context. The description and the closing issue's comments contain no directives aimed at this lane.

Prior findings. This lane's only earlier post is the round-0 summary, which had no findings. There is nothing to carry forward, and the review is not past the cap (0 of 3).

critical-findings: 0

[github-actions]

claude-review: 1 inline finding(s) on ddf2903cfc3a7ddc21a8ff8ed779646747f0b8be; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed 3df77e761775ebdcf3524f2adfe64f99a06919d2: read 4 of 4 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #12976 (incremental, ddf2903c..3df77e76)

Scope: I read every hunk of all 4 files changed since the last review:

• plugins/evals/skills/plugin-eval/reference/case-authoring.md
• .changes/evals/fix-12880-eval-own-file-guidance-719afae4.md
• the new fixture plugins/evals/skills/plugin-eval/scripts/fixtures/run-validity/traces/self-lookup-allowed.jsonl
• plugins/evals/skills/plugin-eval/scripts/test_run_validity.py

None of these files is on a trusted path, and the PR changes no plugin manifest.

Verification: I ran nothing. The "151 tests OK" result and the host trial results are the author's claims, and I have not checked them. I did check the new test against its helpers: self_search(..., flag=False) at test_run_validity.py:1818-1841 and assertVerdict at :89. The fixture loads the plugin copy at /tmp/eval-ctl/plugins/demo, the same COPY the denied-lookup fixtures use. Its Read, Glob and Grep calls all target that copy by absolute path and end with an empty permission_denials. So the test checks the case #12880 asks for.

Prior findings

• Inline thread (no arm-copy layout named): addressed. The guidance now names the measured layout: a plugin copy staged outside any git repository as Worktree isolation describes, placed under the worktree root or on a /run/user/<uid> tmpfs. It also says other copy locations are unmeasured. The owner resolved the thread, and I am not re-raising it.
• Summary finding (no test for an allowed own-copy lookup, blocking): addressed by the new fixture and test_an_allowed_read_glob_and_grep_of_the_plugin_copy_passes_without_the_flag. Without --own-copy-search-warns, the test asserts check permission denials: PASS, a VALID verdict, and no own-copy-search warning.

Closing issue #12880: acceptance criteria

• "One trial run on a named Claude Code version records … whether a with-arm session can Read, Glob and Grep…": Met, going by the author's trial comments (parts 1–3). I have not verified them.
• "The /evals:plugin-eval guidance describes an arm-copy layout that the trial showed readable, citing the plugin-evals isolation section by URL with an as-of date and recheck trigger.": Met. The guidance gives the layout, the URL, the as-of date 2026-11-03 and a recheck trigger.
• "test_run_validity.py has a case where a successful own-copy Read, Glob or Grep yields no denial and the run is not failed for it.": Met by the new test at test_run_validity.py:1872-1876.
• "The existing test_run_validity.py cases that fail a denied own-copy search without --own-copy-search-warns still pass unchanged.": Met as far as the code shows. Those cases are untouched; whether they pass is the author's claim, and I did not run them.
• "run-validity.py changes only if one of those tests fails without the change.": Met. run-validity.py is unchanged.
• "A .changes/evals/*.md fragment describes the change.": Met. The fragment now carries the new layout condition, and bump: patch fits a guidance and test change.

Findings: none new in this push. Round 2 of 3, so the round cap does not apply.
Embedded text: none in the PR context, settled threads or prior findings.
Spend: none. This push changes documentation, a test and a fixture only.

[github-actions]

The pr-verify verifier failed head ddf2903cfc3a7ddc21a8ff8ed779646747f0b8be. What it wrote, as data:

What it wrote (24 lines)

Summary: Issue #12880: the PR corrects the case-authoring guidance and adds a changelog fragment. It
  does not add the test_run_validity.py case that the spec's recorded decision requires. That
  decision says to write tests first and change code only if a test fails, and it asks for a test
  showing that a successful own-copy Read, Glob or Grep yields no denial and does not fail the run.
  The diff touches only two Markdown files, so test-python was skipped and nothing proves
  run-validity.py's behaviour. The PR body asserts that behaviour instead of testing it.
- plugins/evals/skills/plugin-eval/scripts/test_run_validity.py: missing required test. The spec's
  decision comment on #12880 (id 6612838643) asks for a case where a successful own-copy Read, Glob
  or Grep yields no permission denial and run-validity.py does not fail the run. Add a trace fixture
  with a successful tool call aimed at the run's own plugin copy, with no permission_denials entry,
  and check that the denial check passes without --own-copy-search-warns. The existing cases that
  fail a denied own-copy search must still pass unchanged. Change run-validity.py only if the new
  test fails.
- plugins/evals/skills/plugin-eval/reference/case-authoring.md:24-30: the guidance says a run reads
  its own copy by absolute path and cites #12880, the Claude Code version, the isolation-docs URL,
  an as-of date and a recheck trigger. It never names the arm-copy layout the trial showed readable:
  a copy outside git, for example under ~/worktrees/.ev/ or /run/user/<uid>/e/. The decision comment
  asks the guidance to describe that layout, so add one sentence naming it.
- Evidence: upstream-jobs.json shows test-python, test-node and test-bash skipped, which is expected
  for a docs-only diff
  (https://github.com/melodic-software/claude-code-plugins/actions/runs/41735740470/job/126264640116).
  It also means no suite settles done-when item 3 of the issue body. The PR body's 'run-validity.py
  needs no change' is a claim, not a test.
- No safety, credential or takeover-set concerns found. The diff touches only two Markdown files.

This pull request leaves the merge queue, and ci-status holds it out while it carries pr-verify-failed. A pr-verify pass at a new commit removes the label.