fix(evals): eval runs cannot read the plugin copy's own files, so skills that use them measure low

Problem

In a claude plugin eval run, the plugin copy under test sits outside the run's sandbox, so the session cannot Glob or Grep its own skill files, reference docs or context/ files. A real install can. Any skill whose body points at its own supporting files, or whose session looks them up, therefore runs handicapped in every eval run. The arm that does not use that skill does not. A comparison between arms then tilts against the skill.

Seen in #12478's 144-run pass. Own-copy searches appeared only in the arms that run /planning:devils-advocate: a Glob of skills/devils-advocate/** and a Grep of reference/config.md in the plugin copy. B had 2, C 1 and the control 3; the reviewer-only arm had none. The skill links ../../context/problem-framing.md (plugins/planning/skills/devils-advocate/SKILL.md:137). run-validity.py marks such a denial INVALID in a normal arm, and its --own-copy-search-warns flag is meant for the broken side of a negative control only (plugins/evals/skills/plugin-eval/scripts/run-validity.py:13-16). The pass's control failed, and the result was recorded as unmeasurable: #12478 (comment)

Done when

An eval run can read the loaded plugin copy's own files (skills, reference/, context/) from inside the session, as a real install can. This could be the plugin-eval guidance placing the copy where the sandbox allows reads, or an upstream setting if one exists. Either way, cite the upstream docs read.
The /evals:plugin-eval guidance says how to lay out arm copies so a skill's own files stay readable.
run-validity.py keeps failing a real unreadable reference, but no longer fails a run for a readable own-copy lookup once the layout above is used.

Refs: #12478

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none

Understood: eval runs can't read the plugin copy's own files, so skills that rely on them measure low. The fix is guidance on an arm-copy layout that stays readable, plus tests that keep run-validity.py failing a real unreadable reference.

The recorded decision is below. It settles the three open questions, so none are left.

Decision recorded.
Agent choice, not an owner answer.

Answers to the three open questions in the refinement, decided by the Issue Triage session:

Is there a supported way to make the loaded plugin copy readable inside an eval session?
The docs document none. The isolation section says add_dirs "grant read access only" to "Directories inside the case directory" and that "A run can't read the eval directory", which sits inside the plugin. So the first step is one trial run on the current Claude Code version that checks whether a with-arm session can Read, Glob and Grep its own skill files. [...]
[quote truncated; the full decision is at the link]

#12880 (comment)

The trial run needs a live eval, so it goes to the host queue (host-run). It is not a reason to hold the issue.

_{Automated triage, applied by the lanes App.}

Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 1 (a live Claude Code CLI run: "One trial run on a named Claude Code version records, on this issue, whether ..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

Intended outcome: Plugin-eval guidance says how to lay out arm copies so a skill's own files stay readable, if the docs allow it, and run-validity.py stops failing a readable own-copy lookup while still failing a real unreadable reference.

Related work

feat(evals): repair or retire eval cases whose with-arm never loaded the skill #12043 (related): Eval cases whose with-arm never loaded the skill; a different cause of untrustworthy skill-arm results.
perf(evals): cut the plugin-eval skill body through progressive disclosure #12661 (related): Cuts the same plugin-eval SKILL.md body, so layout guidance added here may collide with it.
fix(evals): warn on non-Skill denials in invocation-probe runs #11655 (related): Merged change to run-validity.py denial handling for probe runs, in the same denial-classification area.

Open questions

Is there a supported way to make the loaded plugin copy readable inside an eval session? The plugin-evals docs show only context.add_dirs, which grants read access to directories inside the case directory, and no setting for the plugin copy. Should the guidance lay out arm copies under an add_dirs entry, or document that the sandbox blocks the copy?
RECOMMENDED: Document an add_dirs-based layout only if a trial run shows the session can then Read, Glob and Grep the files. Otherwise record the limit with the docs cited and keep the validity rule below. Basis: https://code.claude.com/docs/en/plugin-evals: add_dirs 'grant read access only'; 'A run can't read the eval directory'; no plugin-copy read setting found
Done-when item 1 needs an observed run to confirm the copy is readable. That is a paid claude plugin eval run an unattended lane cannot verify offline. Should the issue require such a run, or accept guidance plus run-validity.py tests only?
RECOMMENDED: Accept guidance plus tests in test_run_validity.py, and split the paid confirming run into its own owner-run check. Basis: judgment
What should run-validity.py do for an own-copy lookup once the layout is used? Today own_copy_denial only warns under --own-copy-search-warns, and a denial otherwise fails the run. A readable lookup produces no denial, so it may already pass. Is any code change needed beyond tests proving that?
RECOMMENDED: Add tests showing a successful own-copy Read, Glob or Grep raises no denial and a denied read of a missing reference still fails. Change the script only if a test fails. Basis: plugins/evals/skills/plugin-eval/scripts/run-validity.py:1875-1896 and :2058-2080

Sources read

plugins/evals/skills/plugin-eval/scripts/run-validity.py:1875: own_copy_denial treats a Read, Grep or Glob aimed at the run's own plugin copy as a warning only under --own-copy-search-warns
plugins/evals/skills/plugin-eval/scripts/run-validity.py:2058: check_denials passes own-copy denials as warnings only when the flag is set; otherwise they count toward FAIL
plugins/evals/skills/plugin-eval/SKILL.md:130: guidance copies the plugin into a new empty directory for worktree repos; it says nothing about keeping a skill's files readable
plugins/evals/skills/plugin-eval/SKILL.md:300: mentions --own-copy-search-warns for a negative control's broken side
https://code.claude.com/docs/en/plugin-evals: run isolation; add_dirs grants read-only access to directories inside the case directory; no setting for reading the plugin copy

To answer, post a comment that starts with this line:

**Decision recorded.** <answer>

_{Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.}

The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.

No description provided.

Host run: trial, part 1. Lane 2, a WSL host, Claude Code 2.1.317, agent model claude-sonnet-5-5, 2026-11-03.

A minimal plugin ownread with one skill: its body tells the session to Read its own skills/probe/reference/token.md, Glob its own skill directory, and Grep its own context/ctx.md. Each file holds a unique token, so a reply carrying the token proves the tool returned the file. The plugin copy sat outside git at ~/worktrees/.ev/lane-2/12880/ownread (the path on each trace's init line), TMPDIR=~/.ev/l2-12880, --ablation none --runs 1 -j 2 --trust-plugin --keep-temp.

Without a Bash grant (Read, Glob, Grep, Skill only): readable

Case	Tool input	Score
own-read	Read `~/worktrees/.ev/lane-2/12880/ownread/skills/probe/reference/token.md`	1
own-glob	Glob `*/` in `.../ownread/skills/probe`	1
own-grep	Grep `CTXTOK-[0-9A-F]+` in `.../ownread/context/ctx.md`	1

run-validity.py: verdict: VALID, check permission denials: PASS (no denied tool call in 4 traces). The fourth case (adddir-read, a token copy granted through context.add_dirs) scored 0 because the session never called Read; it says nothing about access yet.

With `--allow-tools Bash` (the condition #12478 ran under, per #12835's body): not measured on this host

All 3 runs were refused before the first turn: "the Docker (~/.docker, DOCKER_CONFIG) credential store on this machine holds a symbolic link inside it (at or under its "contexts" entry), so the Bash sandbox cannot reliably exclude it — a Bash-granting evaluation cannot run here". A retry with DOCKER_CONFIG set to an empty directory for that one process was refused the same way. On this host ~/.docker/contexts and ~/.docker/features.json are symbolic links into /mnt/c/Users/<user>/.docker/ (Docker Desktop's WSL integration), dated 2026-11-03 16:05 local, so the refusal started after #12478's pass. 0 model turns were spent on the Bash leg.

What this settles and what it does not

Settled: with no Bash grant, a session reads, globs and greps its own plugin copy by absolute path. The plugin copy is not unreadable as such.
Open: planning: plan_stress_test per mode, settled by the devils-advocate eval #12478 granted Bash, which puts the run under the OS sandbox (plugin-evals: "When you grant Bash in any form, every command runs under Claude Code's OS-level sandbox. Writes are confined to the run's workspace, your home directory and Claude Code configuration are unreadable"). Every plugin copy here sits under the home directory. Whether that is what denied planning: plan_stress_test per mode, settled by the devils-advocate eval #12478's own-copy searches needs the Bash leg, on a host whose Docker store holds no symbolic link (or after Docker Desktop's integration stops linking it).
Also open: the devils-advocate links are relative (../../context/…), so a run may resolve them against the workspace and land outside the plugin. The Bash leg's traces should record the exact tool inputs.

Issue stays open, claimed by lane 2.

Host run: trial, part 2 (Bash granted). Same suite and host as part 1, with --allow-tools Bash, run through the owner-approved plugins/evals/skills/plugin-eval/scripts/docker-links.py run -- … swap (#8378), which replaces Docker Desktop's two ~/.docker links with copies for the run and restores them after. find ~/.docker -mindepth 1 -type l listed 2 links before and 2 after.

Case	Score	Turns
own-read	1	2
own-glob	1	2
own-grep	1	2

run-validity.py: verdict: VALID, check permission denials: PASS (no denied tool call in 3 traces), check plugins loaded: PASS.

Correction to part 1: the Docker-link refusal is not a block on this host, because the swap route above exists. Part 1's retry set DOCKER_CONFIG for one process before I found that route; #8378's decision keeps DOCKER_CONFIG forbidden. That run was refused at start and changed nothing.

Reading

Under the OS sandbox that a Bash grant turns on, a session still reads, globs and greps its own plugin copy by absolute path when the copy sits outside git under ~/worktrees/.ev/. So the issue's premise ("the plugin copy under test sits outside the run's sandbox, so the session cannot Glob or Grep its own skill files") does not reproduce in this layout on Claude Code 2.1.317.

What it leaves open: why #12478's own-copy searches were denied. Their traces are deleted, so the exact tool inputs cannot be read. Two candidates fit the evidence, neither confirmed:

Path form. devils-advocate links ../../context/problem-framing.md (plugins/planning/skills/devils-advocate/SKILL.md:137), and the issue records a Glob of skills/devils-advocate/**; a relative pattern resolved against the run's workspace, not the plugin copy, lands outside both.
Copy location. A copy placed somewhere other than ~/worktrees/.ev/ (for example the /run/user/<uid>/e/ tmpfs option some lanes use) may fall under a different sandbox rule.

Next on this issue: one small trial per candidate (a skill that follows a relative ../../ link and a relative Glob; the same suite with the copy under /run/user/<uid>/e/), then guidance in /evals:plugin-eval for whichever one denies. If neither denies, the done-when items are met as written (own files readable; the layout is the one above; run-validity.py has nothing to stop failing for a readable lookup) and the issue closes on that record.

fix(evals): name the measured plugin-copy layout and pin a readable o…

Part 3 and the open cause. Two more Bash-granted legs (through the #8378 swap), 7 runs, all VALID with no denial: a skill following a relative ../../context/ctx.md link read it (1 of 1 at both copy locations), and the full suite passed with the copy on the /run/user/<uid> tmpfs (5 of 5). A pathless Glob skills/probe/** returned "No files found" (it searched the run's empty workspace), which is a miss, not a denial.

So no layout tried here reproduces #12478's own-copy denials, and their traces are gone, so the cause stays unknown. No re-run of #12478 is planned: its unmeasurable result rests on its negative control not detecting a loss, and its 6 denied runs were already excluded before any score was read. If the denial is real, run-validity.py still fails any denied own-copy lookup by default, so the next run that hits one reports it, and --keep-temp keeps the trace with the exact path. Lane 2's choice, not an owner answer.

Navigation Menu

Repository navigation

fix(evals): eval runs cannot read the plugin copy's own files, so skills that use them measure low #12880

Problem

Done when

3 remaining items

Without a Bash grant (Read, Glob, Grep, Skill only): readable

With `--allow-tools Bash` (the condition #12478 ran under, per #12835's body): not measured on this host

What this settles and what it does not

Reading

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

fix(evals): eval runs cannot read the plugin copy's own files, so skills that use them measure low #12880

Description

Problem

Done when

Activity

github-actions commented on Nov 3, 2026

melodic-automation-lanes commented on Nov 3, 2026

melodic-automation-lanes commented on Nov 3, 2026

3 remaining items

melodic-automation-lanes commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 3, 2026

Without a Bash grant (Read, Glob, Grep, Skill only): readable

With --allow-tools Bash (the condition #12478 ran under, per #12835's body): not measured on this host

What this settles and what it does not

kyle-sexton commented on Nov 3, 2026

Reading

kyle-sexton commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

With `--allow-tools Bash` (the condition #12478 ran under, per #12835's body): not measured on this host