feat(planning): concise skill text measured by evals, plus outcome cases and probes

[kyle-sexton]

Refs: #6552
Refs: #6476

Summary

The planning slice of the concise-skills rollout, run through the per-plugin runbook (/playbooks:skill-authoring). Protocol: before and after on Opus and Sonnet, outcome cases at 3 runs, held-out triggering at 1 run per query, --ablation none, --max-cost-usd 5 on every pass, before and after launched together, Claude Code 2.1.299.

• Bodies: ten of the eleven model-invoked skills are tighter; interview ships its base text (see Verification). The cuts are rationale that changes no action, restated text, and filler. Every gate, route, stop and report line stays.
• Descriptions: audit-answers, brainstorm, design and plan are shorter, with every quoted trigger phrase kept. The other seven are unchanged.
• setup ships unchanged. It is user-invoked and no outcome case reaches it. Its rewrite also dropped every trigger phrase from the description.
• Five new outcome cases (au-, de-, dg-, dh-, qu-) cover audit-answers, design, draft-goal-condition, design-handoff and questionnaire. Every one of the eleven skills now has at least one case.
• Grader fixes across the existing 29 cases, each with must-pass and must-fail samples (see Verification).
• Frozen trigger-probe sets for the eleven skills under probes/, 18 queries each.
• No run artifacts are committed.

Fix

Per-skill decision table

Sizes: description characters, then file lines and words. Line counts rise where long paragraphs were wrapped at 100 columns.

Skill	Description	Body	Decision
audit-answers	498 → 485	127 → 186 lines, 2,159 → 1,922 words	new
brainstorm	487 → 480	47 → 89 lines, 775 → 751 words	new
design	453 → 446	304 → 455 lines, 4,396 → 3,796 words	new
design-handoff	412 → 412	139 → 171 lines, 1,603 → 1,545 words	new
devils-advocate	426 → 426	247 → 326 lines, 3,163 → 2,775 words	new
draft-goal-condition	429 → 429	149 → 237 lines, 2,628 → 2,341 words	new
interview	500 → 500	498 lines, 11,387 words	old (reverted; Opus skill mean fell 0.70 → 0.64)
plan	486 → 468	453 → 380 lines, 9,017 → 8,033 words	new
prd	482 → 482	365 → 314 lines, 4,364 → 3,939 words	new
questionnaire	481 → 481	50 → 86 lines, 855 → 771 words	new
setup	651 → 651	167 lines, 1,549 words	old (no outcome case reaches it)
wayfind	439 → 439	243 → 236 lines, 2,483 → 2,272 words	new

Meaning diff

/docs-hygiene:compress compare ran two fresh labelers per skill, one on Opus and one on Sonnet, against base text extracted from origin/main. Every SEMANTIC LOSS row in the union was restored verbatim before the measured passes:

• interview: the two example override phrases ("this isn't a code task", "interview me on this decision"). The whole skill later went back to base.
• A refine lane commit (a4dff33) escaped the literal em and en dashes in two grader regexes as —/–; regex behavior is unchanged.
• brainstorm: the reason no artifact is persisted by default.
• design-handoff: "a producing model rubber-stamps its own recap", and "Silent gaps are exactly what the binary gate exists to catch".
• design: the /planning:design module argument example, and "criteria changes land in /planning:design-handoff only" (an ambiguity row on both labelers).
• draft-goal-condition: "they feed that shape rather than replace it" (ambiguity row).
• plan: the rationale for pre-drafted flip lines (ambiguity row on both labelers).
• setup: both labelers flagged the dropped trigger phrases, so the whole skill went back to base.

audit-answers, devils-advocate, prd, questionnaire, wayfind and plan passed on both labelers.

Verification

Outcome suite (3 runs per case, frozen graders)

Before freezing, four fresh agents audited every deterministic grader for negations before a name, single matches where several facts are required, shell writes missing redirect, tee, cp, mv, install, dd, sed -i, pipe-to-shell and interpreter forms, reads graded only through Bash, "no X" substrings, and graders too narrow for the skill's own wording. Each fix landed with a must-pass and a must-fail sample. Three Bash-write guards were added (no-bash-write-back, no-bash-code-write, no-bash-plan-write), and two LLM graders were added where a regex could not express the outcome (plan-next-no-detour, navigator-not-guess). The suite was then frozen (digest recorded) and not edited after the first measured run. No run ended in an error.

Skill (cases)	Sonnet before	Sonnet after	Opus before	Opus after
audit-answers (1)	0.93	0.87	0.80	0.77
brainstorm (2)	0.78	0.75	0.69	0.69
design (2)	1.00	1.00	0.88	0.93
design-handoff (1)	0.83	0.83	0.62	0.67
devils-advocate (1)	0.93	1.00	0.80	0.80
draft-goal-condition (1)	0.73	0.87	0.93	0.87
interview (16), measured rewrite	0.76	0.75	0.70	0.64 (reverted to base)
plan (3)	0.80	0.89	0.83	0.84
prd (2)	0.58	0.75	0.75	0.67
questionnaire (1)	1.00	1.00	1.00	1.00
wayfind (4)	0.72	0.75	0.75	0.75

• Means over 34 cases, as measured: Sonnet 0.783 → 0.801 and Opus 0.749 → 0.720, both inside the 0.05 margin. As shipped (interview at its base score): Sonnet 0.783 → 0.806 and Opus 0.749 → 0.747.
• interview reverted to base after the review round. The fresh-context judge traced none of its Opus misses to the rewrite (below), but its Opus skill mean fell 0.70 → 0.64, past the margin at skill level, and the rewrite cut only about 3% of its words. Shipping base text removes the question without a remeasure.
• Review-round grader fix, rerun: dg session-note now requires an effect word on the /clear line and on the resume line ("See the docs for /clear and resumed sessions." is a must-fail sample); the dg case was rerun on both versions and both models, and its row above is the rerun.
• After the measured passes, the pre-ready review found five lenient graders in the four new Bash-granting cases (au verdict-per-answer and limit-challenged, dh verdict-fail, de numbered-recommendations, dg session-note). Each was fixed with the reviewer's strings as must-fail samples, and those four cases were rerun on both versions and both models; the table and means above use the rerun.
• Three cases dropped by a third or more, all on Opus and all interview: final-confirm-active-reply (0.56 → 0.33), quality-bar-asked-bookmark-sync (0.33 → 0.00) and quality-bar-asked-ticket-dashboard (1.00 → 0.67).
• A fresh-context judge read the trace of every failing after-run in those three cases, and in every other case whose after score fell on either model (28 after-runs in all). It compared each against a before-run of the same case. No miss traces to a rewrite hunk:
  • Most were grader artifacts: the run met the expected outcome, but the frozen grader missed the wording. Examples are a quality-bar question asked as "Is this a personal tool, just for you?" with the tiers listed as alternatives, "Reply "confirmed", or confirmed with your answers to 2 and 3", and "passed the character counter at 592 of 4000 characters".
  • The rest were real misses governed by unchanged text, with the same failure mode in before-runs. Examples are the confirmation gate holding back when no ledger is on disk, and a design-handoff FAIL that routes into a /planning:design round as its unchanged FAIL text says.
• Five cases score at or near 0.00 to 0.50 identically on both versions and both models (interview-scope-cut-moscow, wayfind-next-unsure-midflow, interview-recommendation-basis-and-flip, interview-closing-probe-something-else, prd-next-unsure-midflow). They do not separate the versions, and their graders are listed for follow-up below.

Triggering (held-out, not committed)

There are 6 queries per skill for the 11 model-invoked skills, 4 that should trigger and 2 that should not. They were written blind from neutral purpose statements and run once per query.

Model	Before	After
Opus	66/66	66/66
Sonnet	62/66	62/66

brainstorm (4/6 → 5/6) and plan (5/6 → 4/6) moved by one query on Sonnet, so both were rerun at 3 runs per query on Sonnet: brainstorm 13/18 → 14/18 and plan 14/18 → 14/18. Neither loses, so both descriptions ship.

Static checks

• /skill-quality:check over plugins/planning/skills: 12 passed, 0 failed.
• /evals:validate: every case loads with 0 FAIL, and every deterministic grader agrees with all of its samples.
• measure-invocation.sh validate: PASS on all 11 probe files.
• scripts/check-affected.sh: 7 passed, 1 skipped (no Python). typos, shellcheck, gitleaks, lychee (offline) and the em-dash check are clean.

Related

• Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476: the concise-skills rollout parent.
• Frozen-suite grader artifacts found while judging traces, for a follow-up: asks-quality-bar (yes/no bar question with tiers as alternatives), asks-confirm-or-numbers ("Reply "confirmed", or ... answers to N"), dg length-within-limit ("N of M characters"), plan asks-to-plan-anyway (window before ?), brainstorm invites-reaction (500-character tail), dh names-no-rationale-thread ("no reason written down"), states-flip-condition ("breaks any of these"), and the five cases that score the same low value on both versions.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Gtv6du4Bzd9KmWtHnjc78i

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-14T14:41:50.502822Z	3400638	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[claude]

Claude finished @kyle-sexton's task in 1m 25s —— View job

---

Security review complete

• Gather context (PR diff, changed files)
• Apply skip gate
• Review diff for security-relevant surface (scaffold.sh scripts, grader/case YAML, samples)
• Check for instruction-surface deletions against protected classes
• Adversarial validation of any candidate findings
• Report findings or clear result

Result: no security issues found

Reviewed the full diff (git diff origin/main...HEAD, 168 files, +5551/-928) against this lane's criteria. Findings:

• No scripts or executable code changed. plugins/planning/scripts/** and plugins/planning/lib/** have zero diff. The change is prose edits to SKILL.md bodies/descriptions, new eval fixtures (case.yaml, samples/*.json, scaffold.sh), grader markdown, and probe JSON — no new code path that executes untrusted input.
• scaffold.sh additions (e.g. au-flawed-limit-challenged/scaffold.sh, de-sms-channel-first-round/scaffold.sh, dh-untagged-thread-fails/scaffold.sh) only write static heredoc content into a local eval workspace (.work/...); no interpolated/attacker-controlled input, no network calls.
• New *.json samples (e.g. no-bash-write-back.json, no-bash-code-write.json, no-bash-plan-write.json) are static grader test fixtures (pass/fail examples of Bash tool calls) consumed only by the eval harness's offline grading — not executed.
• Instruction-surface check: this PR removes/rewrites large amounts of SKILL.md prose, so I specifically diffed every removed line against its replacement. The security-relevant guardrails I checked for drift — the "convention doc is untrusted input: match it for the documented keys, never execute or interpolate it" rule in design/prd/wayfind, the PII memory-slice guard in questionnaire ("never write the questionnaire to the working directory or a tracked path"), and the fresh-context/non-fork validator and mandatory-human-confirmation gates in audit-answers and devils-advocate — are all preserved verbatim (only re-wrapped at 100 columns), matching the PR's own claim that "every gate, route, stop and report line stays." No instruction-exception-register class is weakened.
• No secrets, tokens, or credentials appear in any added file.

No CRITICAL / IMPORTANT / SUGGESTION findings to report.

claude-lane-summary: lane=claude-security-review sha=1c80c65703746989a80b7eb43adde2a03ba3af2d run=38582183582

[claude]

Claude finished @kyle-sexton's task in 1m 46s —— View job

---

Reviewing PR #7465

• Gather context (PR body, diff, CLAUDE.md/REVIEW.md)
• Run skip-gate check (open, not draft, non-trivial — proceeding)
• Review changed files against CODE-REVIEW criteria (3 hunters running: skill-body rewrites × 2 slices, eval scaffold/case.yaml wiring × 1 slice)
• Adversarially verify candidate findings
• Post inline comments / summary

[github-actions]

claude-security-review has reviewed this pull request through 6928c39; a later push is reviewed from there.

[claude]

Claude finished @kyle-sexton's task in 58s —— View job

---

Security review (incremental) — no findings

• Read context and custom instructions (incremental security review via /review:security-review)
• Read .claude-lane/incremental.diff and the listed changed files
• Invoke /review:security-review per instructions
• Post findings / summary

Scope reviewed (changes since 1c80c65 only, per the lane's incremental diff):

• plugins/planning/evals/au-flawed-limit-challenged/{case.yaml, samples/limit-challenged.json, samples/verdict-per-answer.json}
• plugins/planning/evals/de-sms-channel-first-round/{case.yaml, samples/numbered-recommendations.json}
• plugins/planning/evals/dg-pytest-ruff-condition/{case.yaml, samples/session-note.json}
• plugins/planning/evals/dh-untagged-thread-fails/{case.yaml, samples/verdict-fail.json}
• plugins/planning/skills/interview/SKILL.md

What changed: grader regex patterns gained more negation forms (nobody, nothing, zero, don't/won't/do not, decline phrasing) plus matching must-pass/must-fail samples, and new flags: m/mi settings on the verdict-per-answer and session-note graders. These are eval-harness grading regexes over trusted LLM transcript fixtures, not user-facing or privilege-bearing code; no injection, secret-handling, or access-control surface here. I checked the lengthened patterns for catastrophic backtracking: all repetition is explicitly bounded ({0,8}, {0,30}, {0,200}, etc.) with no nested unbounded quantifiers, so no ReDoS concern.

plugins/planning/skills/interview/SKILL.md relocates the two override-example phrases ("this isn't a code task", "interview me on this decision") from Step 1 into the Purpose section (per the PR description, Step 1 is digest-pinned so the base wording goes back there), while Step 1 keeps a generic "honor any explicit user override" reference. I checked this against the instruction-surface-deletion lens (a diff that narrows/softens a standing instruction): nothing is actually removed — the override behavior and its exact trigger phrases are still present verbatim, just moved to a different section of the same file. No guardrail is weakened, so this isn't a finding under the protected-class register.

No security issues found in this incremental scope.

claude-lane-summary: lane=claude-security-review sha=f9e9ece112cac950d1be92f875ebdadd73b60419 run=38582430591

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 34006388f7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[claude]

Claude finished @melodic-automation-lanes[bot]'s task in 28s —— View job

---

Security review (incremental) — no findings

• Read context and custom instructions (incremental security review via /review:security-review)
• Read .claude-lane/incremental.diff and the listed changed files
• Invoke /review:security-review per instructions
• Post findings / summary

Scope reviewed (changes since f9e9ece only, per the lane's incremental diff):

• plugins/planning/evals/prd-handoff-scope-clear/graders/names-plan-step.md
• plugins/planning/evals/prd-next-unsure-midflow/graders/names-workflow-navigator.md

What changed: both diffs are a single-line, no-semantic-change edit: the literal em-dash (—) and en-dash (–) characters inside two grader regex character classes are replaced with their —/– escape equivalents (matches the PR's own commit fix(planning): escape em/en dashes in two grader regexes, likely to satisfy this repo's em-dash lint check on non-doc files). The character class matches exactly the same code points before and after; no new patterns, no widened/narrowed matching behavior, no new capture groups.

• Not an instruction-surface change: these are eval grader regex fixtures (test-only, offline-graded transcript matching), not CLAUDE.md/AGENTS.md/a skill or agent body/a hook allow-deny list, so the instruction-exception-register lens doesn't apply.
• No ReDoS concern: quantifiers are unchanged and remain bounded ({0,15}, lazy *?), and the escape substitution doesn't alter backtracking behavior since it's the same character set.
• No executable code, no secrets, no injection surface touched.

No security issues found in this incremental scope.

claude-lane-summary: lane=claude-security-review sha=a4dff3332b43c38d7fd170116e2099bd6a89bb88 run=38582830040

[claude]

Claude finished @melodic-automation-lanes[bot]'s task in 2m 51s —— View job

---

Code review in progress

• Gather context (PR diff, changed files, REVIEW.md, standards index)
• Invoke /review:code-review procedure (security scoped out — a claude-security-review lane already runs on this repo)
• Fan out hunters over eval fixtures (au-/de-/dg-/dh- cases, interview/prd/wayfind grader regex tweaks, probe JSON), and over the 11 SKILL.md rewrites for dropped gates/routes/stops
• Adversarially verify candidate findings
• Post inline comments for findings tied to a changed line, summary for the rest

[claude]

Claude finished @kyle-sexton's task in 2m 4s —— View job

---

Security review (incremental) — no findings

• Read context and custom instructions (incremental security review via /review:security-review)
• Read .claude-lane/incremental.diff and the listed changed files
• Invoke /review:security-review per instructions
• Post findings / summary

Scope reviewed (changes since a4dff3332b only, per the lane's incremental diff):

• .changes/planning/feat-6552-planning-concise-skills-5b8e21c4.md
• plugins/planning/evals/dg-pytest-ruff-condition/case.yaml
• plugins/planning/evals/dg-pytest-ruff-condition/samples/session-note.json

What changed:

• The changelog fragment's prose was updated to say interview is reverted to base (matches the PR's own description of that revert) — no code or security surface.
• The session-note grader regex in case.yaml was tightened to require an effect-bearing clause next to each of /clear and resum(e|ed|ing) (one of removes/restores/keeps/persists/... etc.), not just the bare term appearing in the line. This is the fix for the reviewer-flagged gap ("See the docs for /clear and resumed sessions." previously passed with no stated effect).
• samples/session-note.json gained that exact string as a new must-fail sample.

I traced the new pattern against all of its own pass/fail samples by hand (git diff + file contents): /clear removes it / Resuming ... restores it → effect word present after each anchor → pass; "See the docs for /clear and resumed sessions." → no effect word (removes, restores, keeps, etc.) appears on the line → correctly fails. This resolves the P2 Codex finding from the prior review round.

Security assessment:

• Eval-harness grading fixture over an offline transcript, not executed and not reachable by untrusted/attacker-controlled external input (the only "input" is a bounded model response inside the eval harness).
• No nested unbounded quantifiers ((x+)+ style) — each clause is a single [^\n]*/\w* star followed by a fixed alternation, with the only other repetition explicitly bounded ({0,2}). This is linear/low-order-polynomial in line length, not exponential; given the bounded, non-adversarial input source, no practical ReDoS risk.
• No secrets, no new executable code, no instruction-surface (SKILL.md/AGENTS.md) change in this slice.

No security issues found in this incremental scope.

claude-lane-summary: lane=claude-security-review sha=6928c39f83d184875aea8196d5202e550b16e400 run=38584170984

[claude]

Claude finished @kyle-sexton's task in 4m 48s —— View job

---

Reviewing PR #7465

• Gather context (PR diff, changed files, REVIEW.md, custom instructions)
• Hunt findings in the 10 rewritten SKILL.md bodies (verbatim-restoration claims, gates/routes/stops) — no findings survived
• Hunt findings in new/changed eval fixtures (case.yaml graders, scaffold.sh, samples)
• Adversarially verify any remaining candidates
• Post inline comments / final summary

[claude]

Claude finished @kyle-sexton's task in 8m 51s —— View job

---

Reviewing PR #7465

• Gather context (PR body, diff, CLAUDE.md/AGENTS.md, REVIEW.md, custom instructions)
• Invoke /review:code-review and follow it for this PR
• Post inline comments for findings tied to a changed line
• Post summary comment with cross-file findings and wrap-up

Result: no findings

Scope: code-review lane only (correctness, design, conventions, error handling, tests, docs). Security is out of scope here — pr-review-security/claude-security-review already ran on this head and reported no issues, so this lane doesn't re-check that ground.

What I checked:

• Both findings from Codex's earlier review (on commit 3400638) are already resolved at HEAD (6928c39). git diff origin/main...HEAD -- plugins/planning/skills/interview/SKILL.md is empty — the skill is byte-identical to base, so the "ships a regressed rewrite" concern no longer applies. dg-pytest-ruff-condition/case.yaml's session-note grader now requires an effect-bearing clause on both the /clear line and the resume line; I hand-traced it and confirmed "See the docs for /clear and resumed sessions." correctly lands in the must-fail sample at samples/session-note.json:38.
• The 10 rewritten SKILL.md bodies (audit-answers, brainstorm, design, design-handoff, devils-advocate, draft-goal-condition, plan, prd, questionnaire, wayfind): diffed each against origin/main looking for a dropped gate/route/stop/report line, a broken cross-reference, or a reordered step that changes behavior. Found none — the cuts are rationale/filler, consistent with the PR's own claim.
• New eval fixtures (au-flawed-limit-challenged, de-sms-channel-first-round, dg-pytest-ruff-condition, dh-untagged-thread-fails): hand-traced every grader regex (verdict-fail, numbered-recommendations, basis-lines, twilio-contained, goal-line-names-both-checks, length-within-limit, names-no-rationale-thread, plus the au- graders) against every pass/fail sample, including the trickiest negation and ratio-with-thousands-separator cases. All resolve to the verdict the sample file claims — no grader/sample mismatch found.
• scaffold.sh for dh-untagged-thread-fails and de-sms-channel-first-round: each seeds state consistent with its prompt and expected_outcome (untagged open thread + rationale-less resolved thread; email-only module + Twilio brief).
• Spot-checked a few of the one-line interview-*/graders/*.md em/en-dash escape fixes against their samples — no behavior change, matching the security lane's own read.

Author-claimed and not independently reproduced: the before/after eval score table and the 3-run/1-run triggering counts in the PR description — I verified the grader logic is correct, not that re-running the suite produces those exact numbers.

No CRITICAL / IMPORTANT / SUGGESTION findings to report.

claude-lane-summary: lane=claude-review sha=6928c39f83d184875aea8196d5202e550b16e400 run=38584170948

[github-actions]

claude-review has reviewed this pull request through 6928c39; a later push is reviewed from there.