Repository navigation
fix(evals): a deep TMPDIR stops the eval sandbox from starting, and the run still scores #11740
Description
Observed
On a WSL2 host, claude plugin eval runs under docker-links.py run -- env TMPDIR=<dir> claude plugin eval …, the shape plugins/evals/skills/plugin-eval/reference/sandboxes.md "Docker link swap" gives. Every Bash call in the run returned:
Sandbox is required but failed to initialize: Failed to create bridge sockets after 5 attempts. Restart to retry. Running it outside the sandbox (dangerouslyDisableSandbox) is not allowed either.
No case that needs a shell can pass, so a with-versus-without run scores both arms the same and reports nothing about the skill. The exit code stays 0 and the JSON shows only Bash called 0x per grader, so the run looks like a skill failure rather than a harness failure.
TMPDIR=/home/<user>/worktrees/claude-code-plugins/.eval-batch/tmp-probe(63 characters): every Bash call failed as above.TMPDIR=/home/<user>/worktrees/.ev(24 characters), same case, same tree, same run otherwise: Bash ran, and the case scored 1.0 with the plugin and 0.67 without.
Cause, UNVERIFIED: the sandbox's bridge sockets live under TMPDIR, and a deep TMPDIR pushes a Unix socket path past its length limit (108 bytes on Linux).
Done when
- The plugin-eval run step refuses, or warns before any spend, when
TMPDIRis deep enough that the sandbox's sockets cannot be created, or picks a short directory itself. - A run whose sandbox failed to start is reported as a harness failure, not scored as the skill's result.
-
reference/sandboxes.md"Docker link swap" says how short<dir>must be.
Evidence
Five eval trees on 2026-10-31 for #11681, #11675 and #11633 ran with deep TMPDIR paths and measured nothing for shell-needing cases; the rerun with a short path is in progress.
Activity
github-actions commentedon Oct 31, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: when
TMPDIRis long (about 63 characters in your probe), the eval sandbox can't create its bridge sockets, so every Bash call fails. The run still exits 0 and scores, which makes a harness failure look like a skill failure. You ask for three things. Warn or refuse before spend, or pick a short directory. Report a sandbox-start failure as a harness failure. Document the maximum<dir>length insandboxes.md.The cause is marked unverified (the Unix socket path limit). Confirming it is the first step. I found no duplicate among the open issues.
Automated triage, applied by the lanes App.
github-actions commentedon Oct 31, 2026
github-actions commentedon Oct 31, 2026
kyle-sexton commentedon Oct 31, 2026
Data point from lane 1 (melo-lap-001 WSL): TMPDIR
/run/user/1000/ev(18 characters). Across every Bash-grantingclaude plugin evalrun today for #6554 and #6558, 214 kept traces: 0 contain "failed to initialize" or "bridge sockets", and 183 contain real Bash calls. Consistent with the 24-character pass reported here.