Repository navigation
fix(evals,code-tidying): plain prompts miss plugin-eval and audit-dead-code #13298
Description
Slice of #6405 (eval lane findings), from the #12043 zero-fire survey.
Problem
Plain prompts that should load a skill do not:
- evals
plugin-eval: a plain question about a WSL2 sandbox missingsocat, and one aboutclaude plugin evalargument order (target before--allow-tools), fired the skill in 0 of 3 runs each (41841302783, 41841355473). test(evals): name plugin-eval in two cases that never fired it #13245 then named the skill in both prompts so those cases measure content, so no case now tests the plain-prompt route. - code-tidying
audit-dead-code: the plain "find the dead code and delete it" prompt missed the skill;adc-read-only-delete-requestnow names the skill, so the same gap is untested.
Change
- Edit each skill's
descriptionso these requests match it: forplugin-eval, sandbox preflight failures (missingsocat/bubblewrap) andclaude plugin evalargument order; foraudit-dead-code, a request to find and remove unused code. - Add one plain-prompt triggering case per skill (prompt does not name the skill;
skill-firedgrader, armwith), checked byvalidate-cases.py. - Keep the shared listing budget:
skill-quality:checklisting check passes.
Done when
- Both descriptions edited in one PR per plugin, merged.
- Each new triggering case fires its skill in at least 2 of 3 runs on the PR head (VALID run).
- The two existing named-skill cases still pass.
Activity
kyle-sexton commentedon Nov 4, 2026
Claimed by hitl-backlog (desk session); building one PR touching evals and code-tidying.
github-actions commentedon Nov 4, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: the plugin-eval and audit-dead-code skill descriptions don't match plain prompts. Those are the sandbox-preflight (missing socat/bubblewrap) and
claude plugin evalargument-order questions, and "find and remove unused code". The fix is to edit both descriptions and add one plain-promptskill-firedcase per skill. The skill-quality listing-budget check must still pass.The "fires in at least 2 of 3 runs on the PR head" item is a measured run, so it falls outside what the lane can check. The other items can be triaged as normal. Related: #6405 (parent tracker). I found no duplicate among the open issues.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Nov 4, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 2 (eval: "Each new triggering case fires its skill in at least 2 of 3 runs on the PR he..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 4, 2026
The latest triage verdict holds nothing: the issue is at
status: ready.5 remaining items
github-actions commentedon Nov 4, 2026
github-actions commentedon Nov 4, 2026
melodic-automation-lanes commentedon Nov 4, 2026
The work-resolve-issue lane stopped on this issue (
closed-by-open-pr): an open PR the lanes App or a listed human opened already closes the issue, a race with the earlier-work check, not a finding about the issue; while it stays open the next run stops before any model turn.It keeps
agent-ready, and work-unblock starts it again (retry 1 of 2).In the run's words:
Partial work is on the branch
work-resolve-issue/issue-13298-41874084496-1. The next run continues from it while it is not behind the default branch, and otherwise starts fresh.Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41874084496
kyle-sexton commentedon Nov 4, 2026
Host run. Ran the two new plain-prompt triggering cases (3 runs each) and the three named-skill cases (1 run each) with
claude plugin evalon the PR head.What ran (Claude Code 2.1.318, subscription login, capped scope, empty HOME, with-without ablation, on a
git archivecopy of the PR commit):claude plugin eval <copy>/plugins/evals --case trigger-wsl2-missing-socat --runs 3 --allow-tools Read Glob Grep Skill:skill-fired3 of 3 with-arm, 0 of 3 without.run-validity.py --runs 3: VALID (models WARN only).claude plugin eval <copy>/plugins/code-tidying --case adc-plain-delete-request-triggers --runs 3 --scaffold --allow-tools Bash Edit Write:skill-fired3 of 3 with-arm, 0 of 3 without.run-validity.py --runs 3 --scaffolded: VALID (models WARN only).--runs 1:sandbox-wsl2-missing-socatwith 1.00 / without 0.00;target-before-jsonwith 1.00 / without 0.00;adc-read-only-delete-requestwith 1.00 / without 0.50. Each fired its skill.Against the done-when: each new triggering case fired in at least 2 of 3 runs (3 of 3) in a VALID run; the existing named-skill cases still pass. Merge is the remaining item.
PR: #13302
Target labels: add none; remove none.