Repository navigation
feat(bugs): concise skill text measured by evals, plus outcome cases and probes - #7393
Conversation
…and probes Shorter descriptions and bodies for write and scan, and a one-line description and tighter body for setup, per the skill criteria. Adds seven deterministic outcome cases (five wr-, two sc-) and frozen trigger-probe sets for write and scan. Every labeled meaning loss was restored before measurement; outcome cases and held-out triggering show no loss on Opus or Sonnet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yc14cc9FaH96a7wXc46NrW
A fresh-context review showed several deterministic graders accepting
replies that miss the case's outcome: negated phrasing ("I won't decline
to fix this"), fixes in an ordinary ts or unlabeled fence, plain or table
severity lines, bare "Steps:" lists, and a prose evidence mention. Each
grader is tightened, every adversarial string is added as a must-fail
sample, and the legit reply shapes the stored runs use are added as
must-pass samples. An offline regrade of the 72 measured runs and 14
smoke runs scores the same under the new graders.
- sc-planted-bug-dry-run: evidence label needs a separator; new
gate-dispatched (an Agent prompt carrying the gate's refute stance) and
no-zero-findings-claim graders.
- scan SKILL.md: zero findings advance the cursor only on a rotation or
named-lane run, matching the Modes table.
- write SKILL.md: Purpose again says it does not open a PR.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yc14cc9FaH96a7wXc46NrW
|
Claude finished @kyle-sexton's task in 1m 40s —— View job Security review — no findings
Scope reviewed: the diff touches only Instruction-surface deletions: checked each SKILL.md rewrite against the instruction exception register classes. All three rewrites are prose condensations that preserve every security-relevant control, just reworded or relocated into
No rule was dropped; this reads as a pure rewording per the PR's own meaning-diff labeling process. Eval fixtures/probes: Pre-existing gaps (not introduced by this PR): the PR body already discloses and tracks two P4 findings predating this diff (no untrusted-content framing in bugs skills; scan lane cursor/dedupe trusts any issue author) as #7394 — out of scope here, correctly not re-reported. No CRITICAL / IMPORTANT / SUGGESTION findings. claude-lane-summary: lane=claude-security-review sha=110ab6b1928a0aa0f98ea1f5ddb47592210a31fe run=38471195276 |
Review in progress
|
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
claude-security-review has reviewed this pull request through a539456; a later push is reviewed from there. |
|
Claude finished @melodic-automation-lanes[bot]'s task in 32s —— View job Security review complete
ResultThe only change since This is a wording/formatting change to an eval case's grading description — no code path, no No security findings. 0 inline comments posted. claude-lane-summary: lane=claude-security-review sha=b74bbe6541386b2f07b2a08a1f21eb21cf8ed730 run=38471410024 |
Review in progress
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 110ab6b192
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Claude finished @melodic-automation-lanes[bot]'s task in 3m 47s —— View job Review in progress
|
…ders can fail The eval runner removes tools a case does not grant, so with Write and Edit absent from allowed_tools the no-write and no-edit graders passed regardless of what the skill did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yc14cc9FaH96a7wXc46NrW
|
Claude finished @kyle-sexton's task in 39s —— View job Security review complete
ResultThe only change since Both Blast radius check: both cases run in the plugin-eval sandbox against a scaffold-seeded No security findings. 0 inline comments posted. claude-lane-summary: lane=claude-security-review sha=62881bfb283a3e533adcb321f5ae7c0ad7e3675a run=38474649507 |
|
Claude finished @kyle-sexton's task in 3m 53s —— View job Review complete
ResultNo code-review findings. 0 inline comments posted. Scope reviewed: What I checked, beyond the PR's own disclosed verification:
I did not independently re-run No findings cross the high-signal bar for this lane. Security-relevant items (title-injection guard, write-guard trust boundary, the two P4 pre-existing gaps) are correctly left to the security lane, which already reviewed and found nothing. claude-lane-summary: lane=claude-review sha=62881bfb283a3e533adcb321f5ae7c0ad7e3675a run=38474649522 |
|
claude-review has reviewed this pull request through a539456; a later push is reviewed from there. |
…osed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yc14cc9FaH96a7wXc46NrW
…e cases and probes (#7442) Refs: #6549 Refs: #6476 ## Summary This is the `session-flow` slice of the concise-skills rollout, run through the per-plugin runbook (`/playbooks:skill-authoring`). The protocol: before and after on Opus and Sonnet, outcome cases at 3 runs with deterministic graders, held-out triggering at 1 run per query, `--ablation none`, and `--max-cost-usd 5` on every pass. - **Thirteen skills ship tighter text**: `clean-stop`, `find-handoff`, `handoff`, `keep-going`, `orchestrate`, `orient`, `reanchor`, `recall`, `reconcile`, `retro`, `running-retro`, `show-options`, `workflow`. Total SKILL.md words across the plugin drop 41,956 to 40,049 (-4.5%). The cuts are rationale, history notes and lines repeated elsewhere in the same skill. Every gate, stop, command and report line stays. - **Six skills keep their old text**: `audit-friction` and `check` (not rewritten); `audit-sessions` and `continue-in-background` (the rewrite was within 1% of base); `setup` and `tidy-work` (user-invoked, and no outcome case reaches them). - **Sixteen new outcome cases** under `evals/`, plus the three existing `recall-*` cases moved to deterministic graders only. Every grader has must-pass and must-fail samples. - **Frozen trigger-probe sets** for the seventeen model-invoked skills under `probes/`. - No run artifacts are committed. ## Fix ### Per-skill decision table Sizes are description characters, then SKILL.md lines and words. Outcome scores are means over 3 runs, before → after. | Skill | Description | Lines | Words | Decision | Evidence | |---|---|---|---|---|---| | audit-friction | 352 | 191 | 1,525 | **old** (not rewritten) | `af-`: S 0.78 → 0.67, O 0.67 → 0.67 on identical text (grader defect, see gaps) | | audit-sessions | 742 | 223 | 1,945 | **old** (rewrite was -0.4%) | `as-`: 1.00 → 1.00 on both models | | check | 174 | 32 | 236 | **old** (not rewritten) | `ck-`: 1.00 → 1.00 on both models | | clean-stop | 494 → 492 | 171 → 151 | 1,772 → 1,535 | **new** | `cs-`: S 0.93 → 1.00, O 1.00 → 1.00 | | continue-in-background | 457 | 344 | 3,557 | **old** (rewrite was -0.7%) | `cb-`: 1.00 → 1.00 on both models | | find-handoff | 486 → 486 | 353 → 340 | 3,878 → 3,693 | **new** | `fh-`: S 0.67 → 0.72, O 0.83 → 0.83 | | handoff | 451 → 443 | 498 → 487 | 5,454 → 5,351 | **new** | `ho-`: 1.00 → 1.00 on both models | | keep-going | 528 → 521 | 296 → 291 | 2,766 → 2,696 | **new** | `kg-`: S 1.00 → 0.93, O 1.00 → 1.00 | | orchestrate | 448 → 445 | 336 → 328 | 3,788 → 3,657 | **new** | `oc-`: S 1.00 → 1.00, O 0.83 → 1.00 | | orient | 548 → 548 | 155 → 140 | 1,269 → 1,121 | **new, one restore** | `or-`: S 0.83 → 0.61 with the "What this skill does NOT do" section cut (one run re-ran a background job, two did not route recovery to `keep-going`). Restored verbatim; rerun with a fresh before control: S 0.83 → 0.83, O 0.94 → 1.00 | | reanchor | 497 → 481 | 135 → 125 | 1,322 → 1,211 | **new** | `ra-`: 1.00 → 1.00 on both models | | recall | 674 → 668 | 168 → 163 | 1,488 → 1,444 | **new** | `recall-asks`: 1.00 → 1.00 both; `recall-reports`: S 1.00 → 0.83, O 0.92 → 1.00; `recall-states`: S 0.89 → 1.00, O 0.67 → 0.89 | | reconcile | 498 → 497 | 170 → 152 | 1,522 → 1,350 | **new** | `rc-`: 1.00 → 1.00 on both models | | retro | 350 → 337 | 217 → 213 | 1,764 → 1,604 | **new** | `re-`: S 0.40 → 0.40, O 0.53 → 0.47 (both arms fail the same two graders every run, see gaps) | | running-retro | 488 → 481 | 255 → 244 | 2,252 → 2,091 | **new** | `rr-`: S 0.71 → 0.71, O 1.00 → 0.96 | | setup | 519 | 163 | 1,547 | **old** (no outcome case) | Meaning diff only | | show-options | 493 → 483 | 216 → 190 | 2,027 → 1,717 | **new** | `so-`: S 1.00 → 1.00, O 0.75 → 0.75 | | tidy-work | 529 | 127 | 1,321 | **old** (no outcome case) | Meaning diff only | | workflow | 515 → 512 | 258 → 247 | 2,523 → 2,448 | **new** | `wf-`: S 0.67 → 0.67, O 1.00 → 0.78 (traces read: both failing after runs name `session-flow:clean-stop` without the leading slash, so the measured grader misses them; behaviour matches) | ### Meaning diff A fresh-context labeler (`docs-hygiene:compare-labeler`, Opus) compared each rewritten skill directory against base text extracted from `origin/main`. All 17 rewritten skills came back PASS: no SEMANTIC LOSS, AMBIGUITY or EMBEDDED INSTRUCTION rows. RELOCATED rows point to `reference/gather.md`, `orient/reference/native-recap.md`, `retro/reference/native-insights.md` and `orchestrate/context/gotchas.md`, each checked this session. ## Verification All runs on Claude Code 2.1.299, with before and after launched together. Eval spend (list-price estimate): triggering 43.47 USD, outcome 52.10 USD. ### Outcome suite (3 runs each) | Case | Sonnet before | Sonnet after | Opus before | Opus after | |---|---|---|---|---| | af-unknown-token-stops | 0.78 | 0.67 | 0.67 | 0.67 | | as-sweep-only-no-collect | 1.00 | 1.00 | 1.00 | 1.00 | | cb-dirty-tree-blocks-launch | 1.00 | 1.00 | 1.00 | 1.00 | | ck-install-request-not-performed | 1.00 | 1.00 | 1.00 | 1.00 | | cs-env-worktree-not-pruned | 0.93 | 1.00 | 1.00 | 1.00 | | fh-confirms-before-resume | 0.67 | 0.72 | 0.83 | 0.83 | | ho-stops-after-resume-prompt | 1.00 | 1.00 | 1.00 | 1.00 | | kg-missing-goal-asks-first | 1.00 | 0.93 | 1.00 | 1.00 | | oc-worker-export-rails | 1.00 | 1.00 | 0.83 | 1.00 | | or-routes-recovery-writes-nothing (after restore) | 0.83 | 0.83 | 0.94 | 1.00 | | ra-fetches-base-reports-drift | 1.00 | 1.00 | 1.00 | 1.00 | | rc-square-books-no-kill | 1.00 | 1.00 | 1.00 | 1.00 | | re-codify-routes-check-asks-first | 0.40 | 0.40 | 0.53 | 0.47 | | recall-asks-before-other-projects | 1.00 | 1.00 | 1.00 | 1.00 | | recall-reports-tried-kept-reverted | 1.00 | 0.83 | 0.92 | 1.00 | | recall-states-scope-searched | 0.89 | 1.00 | 0.67 | 0.89 | | rr-routes-never-codifies | 0.71 | 0.71 | 1.00 | 0.96 | | so-never-invents-names | 1.00 | 1.00 | 0.75 | 0.75 | | wf-machine-going-away-recommends-clean-stop | 0.67 | 0.67 | 1.00 | 0.78 | Sonnet's `as-` and `cb-` after arms measured the rewrites that were then reverted; Opus ran after the reverts, so its after arm for those two cases is the shipped (base) text. Before measurement, every grader was audited for negations before a name, single matches where several facts are required, missing shell-write bypass forms (`cp`, `mv`, `install`, `dd`, `sed -i`, redirects, pipe-to-shell), forbidden reads, stray "no X" matches, and patterns too narrow for either version's wording. Each fix landed with must-pass and must-fail samples, and the suite was frozen (sha256 digest `1f9b14e16614`) before any run. `validate-cases.py` exits 0. After measurement, review tightened two `recall-*` graders: `recall-asks` adds `no-cross-project-read-tool` (a `Read` of another project's transcript), and `recall-reports` splits `unavailable-source-named` into `unavailable-prs-named` and `unavailable-tracker-named`, so each unavailable source needs its own outcome. The `ho-` fixture's `TODO` comment became `next:` for the comment-marker scan. The new suite digest is `92329b997f4c`. The recall rows above were measured under the earlier graders; they are not re-measured yet (see Gaps). ### Triggering (held-out) 6 queries per skill (4 should trigger, 2 should not), written blind from a neutral purpose statement, 1 run per query, not committed. | Model | Before | After | |---|---|---| | Opus | 101/102 | 102/102 | | Sonnet | 99/102 | 99/102 | Misses: `continue-in-background` pos-val-01 (Opus before, Sonnet both arms), `running-retro` pos-val-04 (Sonnet before), `workflow` pos-val-02 (Sonnet after) and pos-val-04 (Sonnet both arms). ### Gaps - **Measured graders too narrow, not fixed here:** `af-` `rejects-bad-token` misses "`--apply-fixes` isn't an argument this skill accepts" (audit-friction text is identical in both arms); `wf-` `names-clean-stop` misses a bare `session-flow:clean-stop`; `or-` `cites-ttl-decision` misses some 90s/300s phrasings. These would be fixed in a follow-up so the frozen suite stays as measured. - **Recall re-measurement pending:** the two tightened `recall-*` graders need a before/after re-run on Opus and Sonnet. It is blocked on this host: `claude plugin eval` refuses Bash-granting cases because `~/.docker/contexts` is now a symbolic link, and the earlier traces were already removed, so no offline rescore is possible. - **`re-` and `fh-`** fail the confirm-before-acting graders in both arms on most runs: the models act before asking under the case's pressure. The rewrite does not change that. ### Local checks `/skill-quality:check` on the plugin: 19 passed, 0 failed. `typos`, `markdownlint-cli2`, `lychee --offline`, `check-changelog-fragments.sh`, and gitleaks (with three commit-less `.gitleaksignore` entries for the CANARY deploy-token fixture) are clean. ## Related - Parent: #6476. Worked examples: #7385, #7391, #7392, #7393, #7390, #7429. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 participant
Closes #6575
Refs: #6476
Summary
This is the
bugsslice of the concise-skills rollout, run through the per-plugin runbook (/playbooks:skill-authoring,reference/rewrite-a-plugin.md) under the owner-approved reduced protocol: before/after on Opus and Sonnet, outcome cases at 3 runs with deterministic graders, held-out triggering at 1 run per query,--ablation none,--max-cost-usd 5per pass.writeandscanship shorter descriptions and bodies.setup(user-invoked) ships a one-line description and a tighter body.tool_usedgraders with offline samples: five forwrite(wr-) and two forscan(sc-, fixtures written by a scaffold).probes/write.jsonandprobes/scan.json, 19 queries each.Fix
Per-skill decision table
wr-cases equal before and after on both models under the corrected graders (1.00 everywhere); under the graders as frozen, Opus +0.06, Sonnet +0.00 (table below)sc-planted-bug-dry-run1.00 before and after on both modelsdisable-model-invocation: true), so the description never reaches the model listing; one-line summary per the criteria. Meaning diff onlywritegained lines while losing 22% of its words: the old body had long unwrapped paragraphs, now wrapped at 100 columns.check-skillreports this as "body grew vs merge-base".Outcome suite
All runs VALID per
run-validity.py(warnings only: no skill-fired grader on slash-invoked cases; judge model not recorded, as no judge is used). Mean of 3 runs,--ablation none, Claude Code 2.1.296, all passes launched concurrently.Scores above are from the graders as frozen. Noise report, after minus before, paired over the five
wr-cases: Opus +0.06, 95% interval [-0.12, +0.24]; Sonnet +0.00, [-0.12, +0.13]. Both read "not shown non-inferior at margin 0.05": the interval is wide at five cases. Thesc-pair is one case, "n too small to call".Every one of the 13 failing grader-runs, in both versions, was classified by a fresh agent as a grader defect (the reply met the case's expected outcome), and none as a real miss. The
wr-feature-request-routes-awaydrop in the after version was three such replies: each named the request a feature request, wrote no bug report and recommended feature intake, but in wording the grader missed (anenhancementlabel suggestion, a bold**Feature request:**option, and a prose**Severity:**bullet the report-detector mistook for a field). After those graders were fixed (see Verification), an offline regrade of all 60 stored runs scores 1.00 in every cell for both versions, so the suite shows no difference between before and after.Triggering (held-out, 16 queries per skill, 1 run per query, both skills combined)
Meaning diff
Two independent
comparelabelers per skill, one on Opus and one on Sonnet. The union of their SEMANTIC LOSS rows was restored before the measured runs:write: questions follow the table's priority order; "or scope the fix separately"; generalizable consumer gotchas go upstream via an issue.scan: the out-of-scope owners (/testing:audit,/mutation-testing:audit,/code-tidying:tidy,/work-items:scan-todos), now one body line rather than description text; the valid-cursor definition and "take the next lane"; the basename example; the refill-cap rationale; the scope-size heuristic; the architecture qualifier in thecodebase-health:auditroute; the consumer-gotchas upstream line.setup: the "narrow-write setup shape" name.Intended, not restored:
setup's trigger list (user-invoked; triggers do nothing there) andscan's description-level routing to the four owners above (moved to the body; held-out near-misses stayed quiet on both models).Verification
validate-cases.py plugins/bugs/evalsexits 0 (seven expected WARNs: Bash is granted by--allow-tools Bash, because both skills'!pre-computed commands need it).d930d842…); one smoke-pass grader fix (claim-verified-recommendationlabel wording, classified a grader defect by a fresh agent, proved on the observed reply and a reply written to fail) and the Bash grant were made before freezing.declines-or-defers-fix(accepts didn't/doesn't/haven't, rejects "didn't hesitate"),severity-critical(label and level on separate lines),claim-verified-recommendation(more label wordings, still three parts),grounded-in-code(symbolic slice math),recommends-feature-intake(written-up feature request, feature label/type),no-five-field-report(a bold Severity label counts only with a level after it; a## Severityheading now counts). Each has new must-pass samples from the observed replies and must-fail samples written to fail. The offline regrade reproduced every recorded score under the old graders (0 mismatches over 60 runs) before applying the new ones; only the 13 failures changed, all fail to pass.tsfence, a five-field report in a slightly different format, or an evidence label without a separator. Tightened graders:no-code-patch,no-patch,no-five-field-report(both cases),no-invented-steps,placeholder-or-targeted-question,declines-or-defers-fix,recommends-fix-workflow,names-feature-request,recommends-feature-intake(nowmflag),grounded-in-code(drops1-based, which the prompt itself supplies),severity-critical,justification-cites-money,title-not-past-tense,evidence-label,no-verified-finding-presented. Added graders:no-zero-findings-claimand a gate-dispatchtool_used(Agent prompt matchingrefut, min 1) onsc-planted-bug-dry-run. Each of the 47 adversarial strings is a must-fail sample that the old graders accepted and the new ones reject.validate-cases.pyexits 0 (258/258 samples as expected). An offline regrade of the 72 stored measured runs plus 14 smoke runs scores 1.00 in every cell under the new graders, the same as before. Two legit stored replies that a first draft falsely rejected were used to loosenrecommends-feature-intakeandgrounded-in-code. Remaining gaps: a fix written as a bare call with no control flow, a negation after a clause break in the feature-request graders, andgrounded-in-codepassing when the reply quotes the prompt's own slice math.scansays the cursor advances only on a rotation or named-lane run (matching the Modes table);write's Purpose says it does not open a PR.probes/scan.json(frozen) has no should-not-trigger probe for coverage gaps or structural tidying.measure-invocation.sh validate plugins/bugs/probespasses. The held-out sets were written blind from a neutral purpose statement and kept outside the repository; the rewriter's brief forbade reading probes or cases.check-skill.sh plugins/bugs/skills: PASS before and after, 0 errors.!commands, setup's missing Gotchas surface.check-changelog-fragments.sh --checkandcheck-orphaned-fixtures.sh --checkpass; catalog and cheat sheet regenerate with no diff.setupis measured by meaning diff only: itsapplywrites under.claude/andcheckreads plugin config, which the eval sandbox blocks (runbook, "Checks eval files trip").sc-clean-file-no-inventionran in the before smoke pass only (1.00 on both models) and was excluded from the measured passes for budget, by a rule written before any after-version number was read.context/*.md) andreference/config.mdwere not rewritten.run-validity.pyreports the four trigger passes INVALID for permission denials: the emitted cases grant onlySkill, and the denials areGlob/Grep/Readcalls (19 and 20 on Opus, 6 and 4 on Sonnet), the same shape in both versions. Trigger counts are read from theSkillgrader regardless.--ablation none); Haiku and Fable untested.sc-planted-bug-dry-runandsc-clean-file-no-inventionnow grant Write and Edit, so theirno-writeandno-editgraders can fail (the runner removes ungranted tools). A run of these cases needs--allow-toolsto include Write and Edit.--max-cost-usd 5.Related
🤖 Generated with Claude Code
https://claude.ai/code/session_01Yc14cc9FaH96a7wXc46NrW