Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Fixed

- `/code-tidying:audit-dead-code` now loads for a plain "find the dead code and delete it" request; it still only reports, and deletion stays with `/code-tidying:tidy`.
7 changes: 7 additions & 0 deletions .changes/evals/feat-13298-plain-prompt-triggering-5d3a9e17.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Fixed

- `/evals:plugin-eval` now loads, without being named, for a sandbox missing socat or bubblewrap before a run and for `claude plugin eval` argument order around `--allow-tools`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
schema_version: "1.1"
name: adc-plain-delete-request-triggers
description: A plain request to find dead code and delete it, which does not name the skill, fires audit-dead-code. It tests triggering; adc-read-only-delete-request tests read-only behavior once the skill is loaded
tags: [audit-dead-code, triggering, natural-language]
expected_outcome: The audit-dead-code skill fires on the plain find-and-delete request.
context:
scaffold_script: scaffold.sh
execution:
prompt: "Find the dead code in this repository and delete it for me. Just remove the unused functions so I don't have to."
allowed_tools: [Read, Glob, Grep, Skill, Bash, Edit, Write]
max_turns: 20
timeout_seconds: 600
graders:
- name: skill-fired
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:code-tidying:)?audit-dead-code"'
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
{
"pass": [
{
"answer": [
{
"tool": "Skill",
"input": {
"skill": "code-tidying:audit-dead-code"
}
}
]
},
{
"answer": [
{
"tool": "Skill",
"input": {
"skill": "audit-dead-code",
"args": "."
}
}
],
"why": "unqualified name"
}
],
"fail": [
{
"answer": []
},
{
"answer": [
{
"tool": "Skill",
"input": {
"skill": "code-tidying:tidy"
}
}
],
"why": "sibling skill"
},
{
"answer": [
{
"tool": "Skill",
"input": {
"skill": "code-tidying:remove-comments"
}
}
],
"why": "sibling skill"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -euo pipefail
bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/adc-seed.sh" \
adc-util-report.sh.txt bin/report.sh \
adc-util-lib.sh.txt lib/util.sh \
adc-util-readme.md.txt README.md
2 changes: 1 addition & 1 deletion plugins/code-tidying/skills/audit-dead-code/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: "Read-only hunt for dead and unused code across four labeled lanes (Knip for TS/JS, vulture for Python, gopls for Go, grep for shell, C# and files no lane owns), each candidate adjudicated dead, uncertain or alive. Use when asked to 'find dead code', to find what is unused in a repository (exports, functions, orphaned files), or whether anything still calls a symbol. Deleting what it finds is /code-tidying:tidy."
description: "Read-only hunt for dead and unused code across four labeled lanes (Knip for TS/JS, vulture for Python, gopls for Go, grep for shell, C# and files no lane owns), each candidate adjudicated dead, uncertain or alive. Use when asked to 'find dead code', 'find the dead code and delete it', 'remove unused code', to find what is unused in a repository (exports, functions, orphaned files), or whether anything still calls a symbol. A find-and-delete request starts here: this skill finds and reports, and deleting what it finds is /code-tidying:tidy."
argument-hint: "[--max N] [--lane knip|vulture|gopls|grep] [target]"
user-invocable: true
disable-model-invocation: false
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
type: tool_used
tool: Skill
input_match: "\"skill\"\\s*:\\s*\"(?:(?:evals:)?(?:plugin-eval))\""
---
10 changes: 10 additions & 0 deletions plugins/evals/evals/trigger-wsl2-missing-socat/prompt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
description: "Triggering case. A plain question about a WSL2 sandbox missing socat, which does not name the skill, loads evals:plugin-eval. It tests triggering; sandbox-wsl2-missing-socat tests the answer once the skill is loaded."
tags: [preflight, triggering]
runs: 3
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
expected_outcome: "The plugin-eval skill fires before the answer"
---

I'm on WSL2 (Ubuntu); `/proc/version` shows `microsoft-standard-WSL2`. `command -v bwrap` prints `/usr/bin/bwrap` and `command -v socat` prints nothing. One case in my plugin's eval suite lists Bash in allowed_tools and I'll pass --allow-tools Bash. WSL2 has the sandbox, so I'm good to pay for the run, right? Answer in under 150 words.
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
{
"pass": [
{
"answer": [
{
"tool": "Skill",
"input": {
"skill": "evals:plugin-eval",
"args": "the user question"
}
}
],
"why": "qualified name, the shape real runs record"
},
{
"answer": [
{
"tool": "Skill",
"input": {
"skill": "claude-api"
}
},
{
"tool": "Skill",
"input": {
"skill": "evals:plugin-eval"
}
}
],
"why": "after another skill"
},
{
"answer": [
{
"tool": "Skill",
"input": {
"skill": "plugin-eval"
}
}
],
"why": "bare name"
}
],
"fail": [
{
"answer": [],
"why": "no Skill call"
},
{
"answer": [
{
"tool": "Skill",
"input": {
"skill": "claude-api"
}
}
],
"why": "a different plugin's skill"
},
{
"answer": [
{
"tool": "Skill",
"input": {
"skill": "evals:methodology"
}
}
],
"why": "a sibling evals skill whose hub lacks the graded fact"
}
]
}
2 changes: 1 addition & 1 deletion plugins/evals/skills/plugin-eval/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: "Guided practice around the `claude plugin eval` CLI, which runs and scores a plugin's eval suite: preflight, no-spend validation, a cost estimate under the ceiling, the run, its validity verdict, and reading the with-versus-without delta. Use when: planning how a headless `claude -p` session launches eval runs, an eval run's delta, arm scores or validity verdict is in question, including whether it can be shared (load this before answering), 'run my plugin evals', 'plugin eval', 'evaluate this plugin', 'eval my skill', 'does my skill actually fire', 'compare two eval runs', 'did my change make the skill better', 'is this gain real', 'aggregate-result.json', 'eval CI gate', 'can this machine run evals', 'how much will this eval cost', 'Bash refuses claude plugin eval', 'plugin eval blocked in a worktree', 'plugin eval refuses: too many worktrees', 'can plugin eval measure CLAUDE.md or rules'. Not for designing success criteria (/evals:design) or the skill-creator JSON eval format."
description: "Guided practice around `claude plugin eval`, which scores a plugin's eval suite: preflight, no-spend validation, a cost estimate, the run, its validity verdict, and the with-versus-without delta. Use when: planning how a headless `claude -p` session launches eval runs, an eval run's delta, arm scores or validity verdict is in question, including whether it can be shared (load this before answering), 'run my plugin evals', 'plugin eval', 'evaluate this plugin', 'eval my skill', 'does my skill actually fire', 'compare two eval runs', 'did my change make the skill better', 'is this gain real', 'aggregate-result.json', 'eval CI gate', 'can this machine run evals', missing socat or bubblewrap, 'how much will this eval cost', 'Bash refuses claude plugin eval', argument order around `--allow-tools`, 'plugin eval blocked in a worktree', 'plugin eval refuses: too many worktrees', 'can plugin eval measure CLAUDE.md or rules'. Not for designing success criteria (/evals:design) or the skill-creator JSON eval format."
argument-hint: "[preflight|validate|run|read <json>|ci|init] [target]"
user-invocable: true
disable-model-invocation: false
Expand Down
Loading