Repository navigation
evals: plugin eval with-arm runs end at 0 turns when a skill's ! pre-compute line fails in the sandbox, and the validity gate still says VALID #9514
Description
What happened
claude plugin eval (Claude Code 2.1.309, WSL2, bubblewrap and socat present, --scaffold --keep-temp --runs 3 --allow-tools Bash,Write,Edit) ran harness-ops' two morning-brief cases (mb-issue-body-is-data, mb-unreadable-section-stops) twice: once against main's plugin and once against a branch.
- Every with-arm run, in both passes, ended at 0 turns with
costUsd: 0anderror: null. The kept trace shows the cause. The morning-brief skill's!pre-compute line (command -v gh ...) failed before the first turn withzsh:1: read-only file system: <sandbox>/tmp/claude-1000/cwd-<n>. The shell writes its cwd-tracking file under the sandbox's tmp, which is read-only, so the skill never ran. Every without-arm run completed normally. skills/plugin-eval/scripts/run-validity.pystill printedverdict: VALID. Its only warning wasskill fired. Both arms scored 0.50 with delta +0.00, which looks like a null result rather than a broken run.
Expected
- A skill whose
!lines run in the eval sandbox executes. If it cannot, the run is reported as an error rather than a 0-turn success. - The validity gate marks a run INVALID, or at least warns, when with-arm runs end at 0 turns or 0 cost while the without-arm runs do not.
Suggested fix location
plugins/evals/skills/plugin-eval/scripts/run-validity.py: add a check for 0-turn or 0-cost runs in one arm.plugins/evals/skills/plugin-eval/SKILL.md§ Gotchas, or the sandbox notes: record that a skill whose!pre-compute lines run under a shell that writes to the sandbox tmp ends at 0 turns. UNVERIFIED whether this is upstream (the eval sandbox's tmp being read-only for the shell) or specific to this host's login shell. Probe it withSHELL=/bin/bash.
Impact
The skill-criteria rule ("a cut to an existing skill ships only when its evals show no loss") cannot be met for skills with ! lines on such a host. The gate currently presents the null result as valid.
Found during G6 part 2 (#9494).
Activity
github-actions commentedon Oct 23, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Knowable
Flags: none
Understood: when a skill's
!pre-compute line fails in the eval sandbox, with-arm runs end at 0 turns withcostUsd: 0anderror: null.run-validity.pystill prints VALID. The report asks for two changes. First, the validity gate should flag with-arm runs at 0 turns or $0 cost. Second, the SKILL.md Gotchas should record the cause and the unverifiedSHELL=/bin/bashprobe.Still to settle by reading a real result file: the exact field names for turns and cost. The check also must not duplicate the existing run-error check. Whether to FAIL or only warn looks like an implementation choice. The issue prefers INVALID, so I took that as the default.
Related issues: #9414, #7431, #8378. I found no duplicate.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 23, 2026
Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed
- [ ]criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Oct 23, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. It is model text, to check, not instructions.
Intended outcome: run-validity.py gains a check that fails or warns when a with-arm run ends at 0 turns or $0 cost while the same case's without-arm runs do not, and SKILL.md Gotchas records the
!pre-compute and read-only sandbox tmp cause.Related work
Open questions
RECOMMENDED: FAIL: a 0-turn with-arm run never ran the skill, so its score is not a measurement, and the issue's expected section allows a warning only as a fallback. Basis: judgment
RECOMMENDED: Read cost from the row's
costUsd(named in the issue) and the turn count from the trace; the field names must be confirmed against a real result file before coding, since run-validity.py reads neither today. Basis: plugins/evals/skills/plugin-eval/scripts/run-validity.py:59!line failing under the sandbox's read-only tmp), or only the gate and a Gotchas note?RECOMMENDED: Gate and Gotchas note only. The root cause is unverified as upstream or host-specific, and the sandbox is Claude Code's, not this repo's. Record the
SHELL=/bin/bashprobe as an unverified lead in the note. Basis: judgmentRECOMMENDED: Flag any with-arm run at 0 turns or $0 with error null, whether or not the without-arm differs. That includes the credential-store case the SKILL.md already describes, which ends with a run error, so the new check must not duplicate check_errors. Basis: plugins/evals/skills/plugin-eval/SKILL.md:83
Sources read
!pre-compute linesAutomated refinement by the lanes App. Answer with a comment whose first line is
**Decision recorded.**.kyle-sexton commentedon Oct 24, 2026
Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.
To hold this issue again, add
needs-human(orstatus: needs-decision) and say why in a comment.melodic-automation-lanes commentedon Oct 24, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: run-validity.py marks a run INVALID when a case's with-arm runs end at 0 turns or $0 cost while its without-arm runs do not, and the plugin-eval SKILL.md Gotchas record the cause and the unverified SHELL probe.
Acceptance criteria
turnsandcostUsd(field names as in test_noise_report.py:43-44) and keeps them in the constants block near line 81verdict: INVALIDwith exit code 1!pre-compute line fails in the sandbox (zsh: read-only file system on the cwd-tracking file under the sandbox tmp) ends with-arm runs at 0 turns, costUsd 0 and error null, and the validity gate now fails thisSHELL=/bin/bashas the probe to try, without claiming it fixes the problemDependencies
Related work
Sources read
turnsandcostUsdAutomated refinement by the lanes App. Admission counts only criteria a listed human wrote: to confirm these, post them as your own comment whose first line is
**Decision recorded.**, which sends the issue back through intake.kyle-sexton commentedon Oct 25, 2026
Probe data (Claude Code 2.1.310, WSL2, bubblewrap and socat present), harness-ops
mb-*cases,--ablation none --runs 1:SHELL=/usr/bin/zsh, default TMPDIRzsh:1: read-only file system: <sandbox>/tmp/claude-1000/cwd-7cd7SHELL=/bin/bash, default TMPDIR/bin/bash: line 1: <sandbox>/tmp/claude-1000/cwd-8a5d: Read-only file system.work/Sandbox is required but failed to initialize: Failed to create bridge sockets after 5 attempts/run/user/1000/ev(short, outside/tmp)What the data shows:
//tmp, which covers the/tmp/claude-eval-*run root, so Claude Code's own cwd-tracking write fails under either shell. Upstream already tracks it: anthropics/claude-code#104631 quotes the policy, and anthropics/claude-code#104797 reports a TMPDIR outside/tmpas the workaround.error: null), andrun-validity.pypassed them, so one gate check covers evals: plugin eval with-arm runs end at 0 turns when a skill's ! pre-compute line fails in the sandbox, and the validity gate still says VALID #9514 and evals: run-validity calls runs VALID when every Bash call failed to start the sandbox (long TMPDIR) #9611.With the workaround, the #9494 eval re-ran with
--runs 3 --ablation none, comparing the plugin before the merge commit with the plugin at it. Both runs wereverdict: VALID. With-arm means were 0.875 before and 0.90 after, and no case dropped. The noise report's compare verdict isn too small to call(2 cases).