Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
bump: none
---

Eval prompt change only; nothing a plugin user runs changes.
4 changes: 2 additions & 2 deletions plugins/evals/evals/sandbox-wsl2-missing-socat/prompt.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
description: "Hard case. Why hard: The base model treats WSL2 as having a working sandbox and stops there; that WSL2 needs socat as well as bubblewrap, and that a missing one means each Bash-granting run is refused, are this plugin's preflight record."
description: "Hard case. Why hard: The base model treats WSL2 as having a working sandbox and stops there; that WSL2 needs socat as well as bubblewrap, and that a missing one means each Bash-granting run is refused, are this plugin's preflight record. It tests behavior once the skill is loaded, not triggering."
tags: [preflight, hard]
runs: 3
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
expected_outcome: "The answer says the sandbox backend is not usable here because socat is missing, so each run granting Bash would be refused and score 0; install socat, restart Claude Code, and confirm with /sandbox before paying for the run"
---

I'm on WSL2 (Ubuntu); `/proc/version` shows `microsoft-standard-WSL2`. `command -v bwrap` prints `/usr/bin/bwrap` and `command -v socat` prints nothing. One case in my plugin's eval suite lists Bash in allowed_tools and I'll pass --allow-tools Bash. WSL2 has the sandbox, so I'm good to pay for the run, right? Answer in under 150 words.
With the evals:plugin-eval skill: I'm on WSL2 (Ubuntu); `/proc/version` shows `microsoft-standard-WSL2`. `command -v bwrap` prints `/usr/bin/bwrap` and `command -v socat` prints nothing. One case in my plugin's eval suite lists Bash in allowed_tools and I'll pass --allow-tools Bash. WSL2 has the sandbox, so I'm good to pay for the run, right? Answer in under 150 words.
4 changes: 2 additions & 2 deletions plugins/evals/evals/target-before-json/prompt.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
description: "Routine guard (regression-guard). The target goes before list-taking flags such as `--allow-tools`, which otherwise swallow it as a value, leaving the CLI with no target. Kept as a routine guard for a coverage gap, not as evidence of plugin value."
description: "Routine guard (regression-guard). The target goes before list-taking flags such as `--allow-tools`, which otherwise swallow it as a value, leaving the CLI with no target. Kept as a routine guard for a coverage gap, not as evidence of plugin value. It tests behavior once the skill is loaded, not triggering."
tags: [run-mechanics, routine, regression-guard]
runs: 3
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
expected_outcome: "The answer gives the target before `--allow-tools` (`claude plugin eval ./plugins/notes --allow-tools Write Edit`); explaining that `--allow-tools` took `./plugins/notes` as one of its values is welcome"
---

I only wanted my notes plugin's evals, but this started loading every eval suite in my repo: `claude plugin eval --allow-tools Write Edit ./plugins/notes`. What's wrong, and what's the right command? Answer in under 120 words.
With the evals:plugin-eval skill: I only wanted my notes plugin's evals, but this started loading every eval suite in my repo: `claude plugin eval --allow-tools Write Edit ./plugins/notes`. What's wrong, and what's the right command? Answer in under 120 words.
Loading