Repository navigation
feat(writing): concise be-concise body measured by evals, plus outcome cases and probes - #8459
Conversation
…e cases and probes Shorter be-concise body, shipped on non-inferiority against three new outcome cases on opus and sonnet. The description keeps its original text: the shorter one fired on fewer held-out trigger queries. Adds three deterministic outcome cases under evals/ and a frozen trigger-probe set under probes/. Closes #6617 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
claude-review reviewed Incremental review (efc9497 → 23a2586)No findings. The change adds one `` marker at the end of one line in each of two eval prompts:
Out of scope for this incremental pass: all other files in the PR were reviewed at efc9497 and are unchanged. |
|
claude-security-review reviewed Security review (incremental, HEAD 23a2586)No security findings. Scope: the change since efc9497, in
|
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Closes #6617
Refs: #6476
Summary
This is the
writingslice of the concise-skills rollout, run through the per-plugin runbook (/playbooks:skill-authoring,reference/rewrite-a-plugin.md). A cut ships only with eval evidence that nothing was lost.be-concisebody ships.SKILL.mdgoes from 1451 to 1155 words; its frontmatter keys are unchanged.evals/, all graded deterministically (regex), each grader with offline pass and fail samples.probes/, 18 queries.Fix
plugins/writing/skills/be-concise/SKILL.md: body tightened. The reader-model statistics now live only inreference/doctrine.md(unchanged), which already held them; the "say it is one unreplicated 1997 study of 51 users" instruction stays in the body. Every completeness-floor item, the posted-record rule, the semantic-diff guard, the word-count report, every boundary route and all four gotchas stay./docs-hygiene:compress compare, two independent labelers) found five losses one labeler rated material; each was restored before any after-version run: "error" in the description's never-drop list, the posted-record clause in the description, the reason for keeping no destination table, the reason short comments skip the subagent, and theadhd:clarifyexample. The description later reverted to the original anyway (table below)..changes/writing/feat-concise-skills-writing-7590c55a.md, minor bump.Verification
Per-skill decision table
Description: held-out positives that fired (out of 8) and negatives that stayed quiet (out of 8), one run per query. Body: mean outcome-case score over 3 runs. The decision rule, written before any run: after ships only if after >= before on trigger counts, and after >= before - 0.05 on every case, on both models.
bc-pr-body-floorbc-status-point-firstbc-posted-comment-offerpos-val-07fired with the original description and did not with the shorter one, on both models. Sonnet's count fell 3 to 2, and the two-version noise report marks the description "not shown non-inferior at margin 0.05" on both models. Opus gainedpos-val-03andpos-val-05, which does not offset a Sonnet loss under the rule.writing, so the near-miss negatives aimed atdiscipline,docs-hygieneandadhdskills test quietness, not which competitor wins.Grader-miss rulings
None. No case dropped.
Validity
16 measured passes (4 trigger, 12 outcome) plus 3 smoke passes, run one at a time. No pass hit a run error, timeout, or session or usage limit.
run-validity.pyjudged all 15 outcome passes VALID and all 4 trigger passes INVALID, each for sandbox Glob, Grep or Read denials outside the sandbox (1 to 4 per pass, on both versions) and for should-trigger queries that did not fire (the quantity being measured). Under the rule set before the runs, only run errors and timeouts invalidate a pass.Checks on this branch
scripts/check-affected.sh --tests: 7 passed, 2 skipped (no shell, no Python changed)./evals:validateonplugins/writing/evals: exit 0, every sample sorts correctly./skill-quality:check plugins/writing/skills: PASS. Before: 3 warnings (description 986 chars, body 162 lines, numbered list without done-conditions). After: 2 (description length, the same numbered list, which is the four properties, not a procedure). The body-length warning is gone.measure-invocation.sh validateonprobes/: PASS, 0 warnings.Related
songwriting).skills/be-concise/evals/evals.json(the skill-creator suite) is unchanged and was not run.🤖 Generated with Claude Code