[kyle-sexton]

Closes #6596
Refs: #6476

Summary

This is the wizard slice of the concise-skills rollout, run through the per-plugin runbook (/playbooks:skill-authoring, reference/rewrite-a-plugin.md). A cut ships only with eval evidence that nothing was lost.

• No skill text changes. A concise hub-and-spoke rewrite of both skills was measured and is not shipped. Every measured part (two descriptions, two bodies) had at least one pass that /evals:plugin-eval judged INVALID, and the protocol ships only parts measured on valid runs (skill-criteria.md, step 4 and step 6).
• Six new outcome eval cases under evals/, three per skill. They are graded deterministically (regex and tool_used), and each grader has offline pass and fail samples.
• A frozen trigger-probe set under probes/, 16 queries per skill.
• Changelog fragment .changes/wizard/feat-6596-concise-wizard-8d82efc5.md, bump none: no skill text changes, so no release.
• No run artifacts are committed: no held-out trigger set, emitted trigger cases, run results, traces, plugin copies or run scripts.

Fix

• plugins/wizard/evals/: six cases with two fixtures. gen-approval-gate, gen-routes-elevated and gen-stage-table-first cover /wizard:generate. una-handoff-dry-check, una-routes-2fa and una-scope-irreversible cover /wizard:unattended.

• Four grader defects, fixed after the smoke pass and before any measured pass. A fresh agent classified all 19 smoke failures: 15 real, 4 grader defects, 0 case defects. Each fix only widens acceptance. Each adds a must-pass sample of the missed shape and a must-fail sample it must still reject. The validator fails on the old graders and passes on the new ones under both Python and JavaScript.

  • no-wizard-run: a path held in a shell variable (S=.../setup-sms.sh; grep ...) counted as a run.
  • routes-unattended: it missed the unattended skill's own scope step after the Skill call.
  • asks-confirm (two cases): it missed "once you answer and approve the table".
  • secret-resolved-by-script: it missed a Read-Host -AsSecureString prompt.

  The suite was frozen after a re-smoke of the fixed cases and was not edited again.

• plugins/wizard/probes/: the trigger probes the held-out set was drawn from.

Verification

Per-skill decision table

Rule, written before any run: on both models, the after description ships only if it fires on at least as many of 8 held-out positives as before and stays quiet on at least as many of 8 negatives. The after body ships only if every outcome case's mean score over 3 runs is at least before minus 0.05. A validity rule was added before any measured pass was read and applies to both versions: the after text for a part ships only if every pass behind that part, on both models, is VALID under run-validity.py.

Skill	Description	Opus pos b / a	Sonnet pos b / a	Neg quiet (all four)	Body	Case	Opus body b / a	Sonnet body b / a
generate	before (INVALID)	8/8 / 8/8	8/8 / 8/8	8/8	before (INVALID)	gen-approval-gate	1.000 / 1.000	1.000 / 0.933 (INVALID)
						gen-routes-elevated	0.500 / 0.667	1.000 / 1.000
						gen-stage-table-first	0.970 / 0.939	1.000 / 0.970
unattended	before (INVALID)	8/8 / 8/8	8/8 / 8/8	8/8	before (INVALID)	una-handoff-dry-check	0.833 / 1.000 (INVALID)	1.000 / 1.000
						una-routes-2fa	1.000 / 1.000	0.833 / 0.500
						una-scope-irreversible	1.000 / 0.952	1.000 / 0.952

• Descriptions revert on validity only. Every trigger pass fired 8/8 and stayed quiet 8/8 on both versions and both models. Seven of the eight trigger passes were INVALID for sandbox denials. These come from the probes themselves, on both versions:

  • generate neg-val-04 Globs a system log directory (3 of 4 passes);
  • unattended pos-val-05 Globs under the Windows mount (all 4 passes).

  Only the before-version Sonnet generate pass was VALID. At 8/8 the counts could show no gain either way.

• Generate body reverts. The after Sonnet gen-approval-gate pass is INVALID: its with-arm cp of the plugin's template.sh was denied, a risk named before the run. As graded, the same case also drops 0.067, past the margin.

• Unattended body reverts. The after Opus una-handoff-dry-check pass is INVALID: two with-arm Bash denials, on reads of /etc/os-release and the hostname. Both runs still scored 1.00. As graded, Sonnet una-routes-2fa also drops 0.833 to 0.500.

• Two-version noise report (the three cases merged per skill, model and version, margin 0.05). Every pair reads "not shown non-inferior at margin 0.05", and Sonnet unattended also flags the una-routes-2fa case drop. That is no evidence of a gain, and given the grader misses below, no evidence of a loss either.

Skill	Model	With-arm change (after minus before)	95% interval
generate	Opus	+0.05	-0.07 to +0.17
unattended	Opus	+0.04	-0.09 to +0.17
generate	Sonnet	-0.03	-0.07 to +0.01
unattended	Sonnet	-0.13	-0.33 to +0.08

• The Concise skills rollout: visualization (2 skills, 2 model-invoked) #6595 margin pattern recurred. One grader flip in three runs costs 1/(3 x graders). Sonnet gen-approval-gate has 5 graders, so one asks-approval miss costs 0.067 and fails the margin. Opus una-scope-irreversible has 7 graders, so one miss costs 0.048 and passes. The same single flip gets opposite verdicts depending on how many graders a case has. The rule was not changed.
• Not applied: with the grader-miss rulings below, every case clears the margin on both models. Each part reverts on the validity rule alone.

Graders changed after review

The committed eval suite differs from the measured one: measured digest 883b830e6b05c41f2236eacb50a97e18156099e7ff376c55cc8b1dab0f720ebe, committed digest 185b5d6c7de2ce0cf756afb16854b5d27008960731593d7008620b992a65069b. After review (Codex), three graders changed in 51c09230e0:

• gen-approval-gate / asks-approval now accepts "reply with your approval" asks.
• The una-routes-2fa and gen-routes-elevated routing graders now grade the Skill call to the target skill (tool_used, arm: both), not the reply text.

ce3f5ac2da restates those two cases' expected_outcome to match. No decision moves, because this PR ships no skill text (every part reverted on the validity rule). The corrected graders serve the re-measure in #11628.

Grader-miss rulings

These were read from the kept traces. Each run meets the case's expected_outcome, but its grader marked it a failure. The rulings are listed and not applied; every score above is as graded.

• gen-routes-elevated / routes-unattended, Opus: 3 before and 2 after with-arm runs called Skill wizard:unattended, planned an admin pwsh script and wrote no .sh. The regex accepts only replies that say "unattended" or "result file ... I read".
• una-routes-2fa / routes-generate, Sonnet: all 3 after runs and 1 before run called Skill wizard:generate, showed the generate stage table and wrote no .ps1. The reply text never names wizard:generate, so the text-only regex fails. This explains the 0.833 to 0.500 drop.
• gen-approval-gate / asks-approval, Sonnet after: "Reply with your approval and I'll run chmod +x" asks for approval. The regex accepts "reply approved" but not "reply with your approval".
• gen-stage-table-first / asks-confirm, Opus after: "Once you've answered these and confirmed or edited the table, I'll write the script" asks for confirmation. Two other misses in this case gate the script on open questions without asking to confirm the table; those failures stand.
• gen-stage-table-first / no-env-grep, Sonnet after (borderline): the match was grep -v '^./.env:', a filter that drops .env lines after a recursive grep. No value appeared in the reply.

The case-drop restore re-run was not run. Every part already reverts on validity, so no restore result could change a decision.

Validity

• Totals. 40 measured passes: 8 trigger, plus 24 outcome sub-passes (each outcome pass split into one sub-pass per case, identically for both versions, to stay under the 5 USD ceiling). Also a smoke pass, a re-smoke of the four fixed cases, and 7 discarded passes listed below. All ran one at a time, with no run error, timeout, usage limit or partial pass among the 40.
• run-validity.py verdicts:
  • 22 of 24 outcome sub-passes VALID. The 2 INVALID are named in the table.
  • 1 of 8 trigger passes VALID. The other 7 were INVALID for sandbox Glob or Read denials outside the workspace: 1 to 6 per pass, on both versions, one of them a Glob of the plugin copy's own root.
• Discarded passes, on both versions alike.
  • A link in the Docker credential store reappeared during the first opus gen-approval-gate before pass. The CLI then refused every Bash-granting run. That pass (two errored runs, INVALID) and five more refused sub-passes (0 turns, cost 0) were set aside.
  • The trigger passes then ran ahead of the outcome passes. Descriptions are decided only from trigger passes and bodies only from outcome passes, so the order change moves neither decision.
  • The outcome passes ran in full once the link was gone.

Unmeasured

• Two grader gaps found in the smoke pass were left unfixed, because fixing them would tighten rather than widen the graders:
  • no-wizard-run misses a script -qec launch;
  • no-paste-back accepts "send me result-latest.json and the transcript".
• The routing cases grade reply text only, so routing done through the Skill call scores as a failure on both versions.
• Whether the with-arm runs reached the rewrite's three unattended spokes was not checked.
• Trigger counts sat at 8/8 on every pass, so they could not show a gain.

Checks on this branch

• scripts/check-affected.sh --tests: 14 passed, 2 skipped.
• /evals:validate on plugins/wizard/evals: exit 0; every sample sorts correctly. The only warnings are the expected allowed_tools grant notes.
• /skill-quality:check plugins/wizard/skills: PASS for both skills, 0 errors, 2 size warnings each. These are unchanged from the base.
• plugins/wizard/skills/generate/template.test.sh: 335 cases, 0 failures.
• plugins/wizard/skills/unattended/template.test.sh: OK.
• scripts/check-changed-skills.sh origin/main: no changed skills.
• scripts/check-changelog-fragments.sh --check-required origin/main: pass.
• markdownlint on the changed Markdown: 0 issues.
• Diff scanned for secrets and machine-specific paths: none found.

Left out: the case-drop restore re-run (every part already reverts on validity); the two tightening grader gaps listed under Unmeasured; spoke reach in the with-arm runs.

Related

• Umbrella: Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476 (concise-skills rollout). Sibling: Concise skills rollout: visualization (2 skills, 2 model-invoked) #6595.
• Validity protocol tracking: fix(evals): let trigger-measurement probe runs pass run-validity #11618, test(visualization): fix the frozen suite's graders and re-measure present and visualize with VALID runs #11628.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VcuponPYh1riXtUL5WmMiY

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-31T19:04:38.085389Z	c3d940e	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-review reviewed ce3f5ac2da20c2530752daa455aeeb8b03aa8055: 0 finding(s), 0 as inline comments.

Incremental review of ce3f5ac2 (changes since 51c09230): no findings.

The change rewords expected_outcome in two eval prompts so each states routing as a Skill call:

• plugins/wizard/evals/gen-routes-elevated/prompt.md now says the run routes to /wizard:unattended by invoking that skill.
• plugins/wizard/evals/una-routes-2fa/prompt.md now says the run routes to /wizard:generate by invoking that skill.

I checked both against their case.yaml graders, which are tool_used on Skill with input_match for unattended and generate. Both also keep the no-.sh and no-.ps1 Write checks. Each reworded text matches its grader, and the PR description describes this exact change, so the wording and the graders now agree. Only the prompt front matter changed. The graders, tool allow-lists and prompt bodies are untouched.

I read the two prompt files, their case.yaml files and the PR context. I did not run the evals.

Trusted-path scrutiny: neither changed file matches a trusted-path glob.

Cost: the change adds no billed resource and no new model or run spend, so I set no hold.

[github-actions]

claude-security-review reviewed ce3f5ac2da20c2530752daa455aeeb8b03aa8055: 0 finding(s), 0 as inline comments.

No security findings in the incremental change since 51c0923.

The change rewrites one expected_outcome line in each of two eval prompts: plugins/wizard/evals/gen-routes-elevated/prompt.md and plugins/wizard/evals/una-routes-2fa/prompt.md. It now says the run routes by invoking the target skill, where before it said the reply points to it. The files are grader text for the eval harness. They aren't standing instructions, hooks, manifests, workflows, or trust controls, and neither matches a trusted path. The allowed_tools lists are unchanged, as are the prompt bodies and the tools granted, so nothing widens.

The skip-gate 2 item applies: no security-relevant surface. The edit doesn't delete, narrow, or soften any standing instruction.

Not run: the criteria file was not read in full, and the adversarial validation step was skipped because there were no candidates.

critical-findings: 0

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c3d940ec24

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".