feat(evals): repair or retire eval cases whose with-arm never loaded the skill

Blocked by: #12032

Outcome

#6405's "Later slices" list "author or repair cases for skills whose suite measured nothing (skill not loaded, grader misreading the run shape)". A case whose with-arm never loads the skill reports a delta that measures nothing. Once the health line answers on dispatched runs, each case it reports as skill loaded: no, or as firing in 0 runs, is either repaired so that the skill loads, or removed with a reason.

Done when

A comment on feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405 lists every case that a dispatched run's health line (.github/scripts/plugin-eval/summarize.mjs:140-142) reports as skill loaded: no, or as firing in 0 runs, and links the run.
For each listed case, a PR either repairs it so that the next dispatch reads skill loaded: yes, or removes it with the reason in the PR body. A repair might be a missing tool grant, a prompt that does not trigger the skill, or a grader that misreads a slash-invoked run.
Each repaired case passes /evals:validate (validate-cases.py) on its suite with no FAIL.
A re-dispatch after the PRs merge shows no listed case still reading no. It is linked on feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405.

Out of scope

Judge calibration (shipped under feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879).
The background hill-climb, which is its own slice of feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405.

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: #12032

Understood: once the plugin eval health line reports on dispatched runs, list every case showing skill loaded: no or firing in 0 runs, then repair or remove each with a reason, validate repaired suites, and re-dispatch to confirm. The issue states it is blocked by #12032, which is still open. Related but not duplicate: #12000-range suite-specific issues such as the vision-case one (run-e2e never fires) cover single suites, while this one covers the fleet-wide sweep.

_{Automated triage, applied by the lanes App.}

Not admitted to the issue lane yet: blocker #12032 is not closed by a listed human or a merge.

No description provided.

The work-resolve-issue lane stopped partway through this issue and needs a person.

Reason: security

Matched security word: token. A match holds the stop whether or not the text names a risk, so read the word in context below.

Why, in the run's own words:

host-only: the work starts from a list of cases that dispatched eval runs report as 'skill loaded: no' or as firing in 0 runs. Building that list means hand-dispatching .github/workflows/eval-browser-dispatch.yml (workflow_dispatch only, one plugin directory per run, paid model calls on the role's subscription token) for each plugin's suite, then reading each job log, which needs a sign-in. This lane can do neither, and it cannot post the required comment on ＃6405 or re-dispatch after the repairs merge. The one dispatched run since the blocker ＃12032 closed, 41353703294, reports one case, pe-ci-pin-full-ids: skill loaded: yes; fired in 1 of 1 runs. That leaves nothing to repair until someone runs the other suites.

What the run says it did not do:

Done-when 1: the ＃6405 comment listing every case that reads 'skill loaded: no' or fires in 0 runs. This needs hand-dispatched eval runs for each plugin suite, the ability to read their sign-in-gated logs, and a way to post on ＃6405. This lane has none of these.
Done-when 2 and 3: repairing or removing each listed case and passing validate-cases.py. Not started, because the case list from done-when 1 does not exist yet.
Done-when 4: a re-dispatch after the repairs merge. This needs a host run.
Tidy, review, verify and checks steps: skipped because the run stopped before any edit and the working tree is unchanged.

Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/41354400109

To retry, remove needs-human and add agent-ready again.

Claimed by a host-run session. This host is surveying the dispatched eval health lines (eval-browser-dispatch.yml, deterministic-only, runs 1, one plugin per run) for the suites #6405 names, then repairing or retiring each case that reads skill loaded: no or fires in 0 runs, on branch host-run/12043-eval-skill-load. A record follows here when the survey ends.

This was generated by AI.

Finding: /source-control:commit does not fire on plain commit requests.

This came out of #12666's eval rounds (PR #12704). On main, with an unchanged copy and Opus on Claude Code 2.1.317, these commit-* cases' with-arm never loaded the skill:

Case	Fired on main
`commit-hook-rejection-no-bypass`	0 of 8 runs
`commit-never-stage-everything`	0 of 8 runs
`commit-no-push-no-pr`	missed on the before arm
`commit-trailer-model-from-session`	1 miss of 12

In those traces the model ran git add and a heredoc git commit -F - itself and never called the Skill tool. That skips the skill's surgical staging and trailer rules.

PR #12704 then named the skill in those four prompts ("... with the source-control:commit skill"). The rounds became VALID, but those cases now test behavior once loaded, not triggering. So this is still open as a triggering defect, not fixed.

Next step for this issue: run /skill-quality:check measure-invocation on /source-control:commit with plain commit prompts (the original wording of the four cases), then change the description and re-measure.

Target labels: add none; remove none.

Evidence for the security stop above: no secret or credential found.

The stop matched the word token. I checked this issue's body and every comment, plus the bodies and comments of #6405 and #12032, for secret-shaped values: Anthropic keys (sk-ant-), GitHub tokens (ghp_-style and github_pat_), AWS access keys, PEM private keys, JWTs and Slack tokens. All three returned 0 matches. The word appears only in the phrase "the role's subscription token", which names a credential without containing one.

needs-human stays on for now, until it is settled whether this stop falls under a recorded owner rule. Host-run work on this issue continues meanwhile.

Decision recorded.
(agent; not an owner decision)

A case built to make no skill call reads "fired in 0" on the health line, and that is its pass. Such cases are listed on #6405 for completeness, but they are neither repaired nor removed: stays-quiet and near-miss cases, a fixture check whose allowed_tools hold no Skill, and a prerequisite-notice case. Done-when 2 and 4 apply only to cases built to fire a skill.

Basis: this issue's Outcome asks for cases whose with-arm "reports a delta that measures nothing". A stays-quiet case's delta measures exactly whether the skill stays out. Each listed case's own expected_outcome says "makes no Skill call", or its allowed_tools exclude Skill (plugins/developer-experience/evals/fx-seed-every-state/prompt.md). Removing them would drop the suite's only false-trigger checks.

Owner decision on Q1, 2026-11-04 ~03:2xZ, Automation Lanes session 19e33801, AskUserQuestion pick (verbatim): "Yes, release them (Recommended)".

The question asked: let the security-word stop release technical senses of a secret noun (model or subscription tokens, per-token measures, token spending, a token's scope, "has no credentials"), while still stopping for a secret value, an exposure, a credential path, and OAuth or session tokens. Owner decision Q4 otherwise stands. #13034 builds it; a fresh security finding on its head (PRRT_kwDOTCGFQM6zbPDZ, CRED_END continuations) is being fixed before merge.

needs-human released.

The security-word stop on this issue matched only the noun phrase "the role's subscription token", and the 01:06Z scan found no secret-shaped value here or on #6405 and #12032. The owner's Q1 answer, quoted verbatim in the 03:16:42Z comment ("Yes, release them (Recommended)"), releases technical senses of a secret noun, model or subscription tokens by name. The release applies now; #13034 only automates it. The GitHub Support hub confirmed this is the Q1 release, not the residual Q4 rule.

Next: drafts #13037 and #13062 go ready.

Claimed by a host-run session for the remaining survey slices: code-tidying, source-control outside p* and commit-*, and planning (deterministic-only health lines). Each case built to fire a skill that reads skill loaded: no or fires in 0 runs is repaired or retired in one PR on branch host-run/12043-survey-rest. Stays-quiet cases are listed, not repaired, per the 02:01Z decision record. A record follows here when the survey ends.

Host run. Three deterministic eval-browser-dispatch runs on main covered code-tidying, source-control's worktree-* cases and four planning tags. One case was repaired.

Each run used deterministic-only, runs 1 and allow-tools=Bash,Write,Edit, on Claude Code 2.1.318:
- code-tidying, tags natural-language natural: 41810568160. 5 of 6 fired.
- source-control, tag worktree: 41824991484. 8 of 8 fired.
- planning, tags draft-goal-condition audit-answers design-handoff questionnaire: 41825160353. 9 of 9 fired.
No line reads skill loaded: no. One case fired in 0: code-tidying adc-read-only-delete-request, 0 of 1.
Repair: test(code-tidying): name audit-dead-code in adc-read-only-delete-request #13204 names code-tidying:audit-dead-code in that case's prompt, so the case now tests behavior once the skill is loaded, not triggering. validate-cases.py gives 0 FAIL, and scripts/check-affected.sh --tests passed 14 and skipped 3.
Triggering finding, not fixed: the plain prompt "Find the dead code in this repository and delete it for me" misses /code-tidying:audit-dead-code.
Survey list, including the cases not dispatched and why: feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405 (comment)
Still unsurveyed: source-control pull-request-ready-hands-pr-to-explain-change, planning tags wayfind devils-advocate scale (their prompts name the skill), and evals outside de-*.

Done-when status:

Items 2 and 3 hold for every case listed.
Item 1 holds for every dispatched case. The slash-invoked cases were not dispatched; their basis is the slice 3 evidence.
Item 4, a re-dispatch after test(code-tidying): name audit-dead-code in adc-read-only-delete-request #13204, test(developer-experience): slash-invoke the skill on the S4 and S7 turn-2 cases #13037 and test(evals,source-control): name the skill in two cases whose with-arm never fired it #13062, is still to do.

Target labels: add none; remove none.

test(code-tidying): drop natural-language tag from adc-read-only-dele…

Host run. Done-when item 4 holds: a re-dispatch after #13037, #13062 and #13204 merged shows every repaired case reading skill loaded: yes, linked on #6405: #6405 (comment posted alongside this one).

Five runs, all skill loaded: yes, fired 1 of 1: 41835891038, 41836049281, 41836255452, 41836309464, 41836413162.
Next: survey the cases the last record left unsurveyed (source-control pull-request-ready-hands-pr-to-explain-change, planning tags wayfind devils-advocate scale, evals cases outside de-*). Any of them that reads no or fires in 0 gets a repair before this issue closes.

Target labels: add none; remove none.

Decision recorded.
(agent; not an owner decision)

Done-when items 1 and 2 cover cases that are meant to fire. A case built not to fire, which reads skill loaded: yes and fires in 0 runs, is working as designed, and it is neither repaired nor removed.

Basis: the four control-* cases hold a tool_used Skill grader with min: 0 and max: 0 over the evals skills (plugins/evals/evals/control-no-trigger/graders/no-skill-fired.md:1-8). plan-proactive-one-sentence-skip expects no plan. route-hillclimb-existing-suite's correct route is outside the plugin. no-model-in-the-loop has no skill-fired grader. Naming a skill in any of these prompts would reverse what it tests. The issue's own repair list ("a missing tool grant, a prompt that does not trigger the skill, or a grader that misreads a slash-invoked run") names defects, and a 0-fire negative control is not one.

Done. Every done-when item holds.

Item 1: feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405 lists every case a dispatched run reported as firing in 0 runs, with its run. The survey of the cases left over: feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405 (comment posted with this close). The earlier slices' lists are in issuecomment-6630218215 and issuecomment-6631661843 on feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405.
Item 2: each case that should fire is repaired: test(developer-experience): slash-invoke the skill on the S4 and S7 turn-2 cases #13037, test(evals,source-control): name the skill in two cases whose with-arm never fired it #13062, test(code-tidying): name audit-dead-code in adc-read-only-delete-request #13204, test(evals): name plugin-eval in two cases that never fired it #13245. Cases built not to fire are out of scope under the decision record above.
Item 3: validate-cases.py gave 0 FAIL on each repaired suite (PR bodies).
Item 4: post-merge re-dispatches read skill loaded: yes and fire for every repaired case: 41835891038, 41836049281, 41836255452, 41836309464, 41836413162, 41845248493 (3 of 3), 41845303504 (3 of 3).

Triggering findings, not fixed: three plain prompts miss their skill, and interval-setting-scope fires 1 of 3 (listed on #6405).

This was generated by AI.

Navigation Menu

Repository navigation

feat(evals): repair or retire eval cases whose with-arm never loaded the skill #12043

Outcome

Done when

Out of scope

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

feat(evals): repair or retire eval cases whose with-arm never loaded the skill #12043

Description

Outcome

Done when

Out of scope

Activity

github-actions commented on Nov 1, 2026

melodic-automation-lanes commented on Nov 1, 2026

github-actions commented on Nov 1, 2026

melodic-automation-lanes commented on Nov 1, 2026

kyle-sexton commented on Nov 1, 2026

kyle-sexton commented on Nov 3, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions