Skip to content

feat(writing): concise be-concise body measured by evals, plus outcome cases and probes - #8459

Merged
kyle-sexton merged 3 commits into
mainfrom
feat/concise-skills-writing
Oct 20, 2026
Merged

kyle-sexton merged 3 commits into
mainfrom
feat/concise-skills-writing

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Closes #6617
Refs: #6476

Summary

This is the writing slice of the concise-skills rollout, run through the per-plugin runbook (/playbooks:skill-authoring, reference/rewrite-a-plugin.md). A cut ships only with eval evidence that nothing was lost.

  • The shorter be-concise body ships. SKILL.md goes from 1451 to 1155 words; its frontmatter keys are unchanged.
  • The original description stays. The shorter description fired on one fewer held-out request on both models.
  • Three new outcome eval cases under evals/, all graded deterministically (regex), each grader with offline pass and fail samples.
  • A frozen trigger-probe set under probes/, 18 queries.
  • No run artifacts are committed: no held-out trigger set, emitted trigger cases, run results, traces, plugin copies or run scripts.

Fix

  • plugins/writing/skills/be-concise/SKILL.md: body tightened. The reader-model statistics now live only in reference/doctrine.md (unchanged), which already held them; the "say it is one unreplicated 1997 study of 51 users" instruction stays in the body. Every completeness-floor item, the posted-record rule, the semantic-diff guard, the word-count report, every boundary route and all four gotchas stay.
  • The meaning diff (/docs-hygiene:compress compare, two independent labelers) found five losses one labeler rated material; each was restored before any after-version run: "error" in the description's never-drop list, the posted-record clause in the description, the reason for keeping no destination table, the reason short comments skip the subagent, and the adhd:clarify example. The description later reverted to the original anyway (table below).
  • Changelog fragment .changes/writing/feat-concise-skills-writing-7590c55a.md, minor bump.

Verification

Per-skill decision table

Description: held-out positives that fired (out of 8) and negatives that stayed quiet (out of 8), one run per query. Body: mean outcome-case score over 3 runs. The decision rule, written before any run: after ships only if after >= before on trigger counts, and after >= before - 0.05 on every case, on both models.

Skill Description Opus pos b / a Sonnet pos b / a Neg quiet (all four) Body Case Opus body b / a Sonnet body b / a
be-concise before 6/8 / 7/8 3/8 / 2/8 8/8 after bc-pr-body-floor 1.000 / 1.000 1.000 / 1.000
bc-status-point-first 1.000 / 1.000 1.000 / 1.000
bc-posted-comment-offer 1.000 / 1.000 1.000 / 1.000
  • Description reverts: held-out pos-val-07 fired with the original description and did not with the shorter one, on both models. Sonnet's count fell 3 to 2, and the two-version noise report marks the description "not shown non-inferior at margin 0.05" on both models. Opus gained pos-val-03 and pos-val-05, which does not offset a Sonnet loss under the rule.
  • Body ships: every case scored 1.000 in all 3 runs on both versions and both models. The noise report calls each single-case comparison "n too small to call", so the evidence is no-drop, not superiority. All three cases sit at the ceiling on both versions, so they show no loss on the measured edges and cannot detect a loss elsewhere; the meaning diff covers the rest.
  • The plugin eval loads only writing, so the near-miss negatives aimed at discipline, docs-hygiene and adhd skills test quietness, not which competitor wins.

Grader-miss rulings

None. No case dropped.

Validity

16 measured passes (4 trigger, 12 outcome) plus 3 smoke passes, run one at a time. No pass hit a run error, timeout, or session or usage limit. run-validity.py judged all 15 outcome passes VALID and all 4 trigger passes INVALID, each for sandbox Glob, Grep or Read denials outside the sandbox (1 to 4 per pass, on both versions) and for should-trigger queries that did not fire (the quantity being measured). Under the rule set before the runs, only run errors and timeouts invalidate a pass.

Checks on this branch

  • scripts/check-affected.sh --tests: 7 passed, 2 skipped (no shell, no Python changed).
  • /evals:validate on plugins/writing/evals: exit 0, every sample sorts correctly.
  • /skill-quality:check plugins/writing/skills: PASS. Before: 3 warnings (description 986 chars, body 162 lines, numbered list without done-conditions). After: 2 (description length, the same numbered list, which is the four properties, not a procedure). The body-length warning is gone.
  • measure-invocation.sh validate on probes/: PASS, 0 warnings.
  • markdownlint on the changed Markdown: 0 issues.
  • Diff scanned for secrets and machine-specific paths: none found.

Related

🤖 Generated with Claude Code

…e cases and probes

Shorter be-concise body, shipped on non-inferiority against three new
outcome cases on opus and sonnet. The description keeps its original
text: the shorter one fired on fewer held-out trigger queries.

Adds three deterministic outcome cases under evals/ and a frozen
trigger-probe set under probes/.

Closes #6617

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 20, 2026 05:00
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T05:02:16.968929Z efc9497 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 23a258658d3e10bb994d80504a7d69ad498c6d2f: 0 finding(s), 0 as inline comments.

Incremental review (efc9497 → 23a2586)

No findings.

The change adds one `` marker at the end of one line in each of two eval prompts: plugins/writing/evals/bc-pr-body-floor/prompt.md and `plugins/writing/evals/bc-status-point-first/prompt.md`.

  • Marker syntax: the form matches the in-file opt-out documented in plugins/ai-slop/README.md:89-92 (a line-level marker with : reason). The lines it sits on are the deliberately wordy draft text, so the exemption covers what it is meant to cover.
  • Graders: I read both case.yaml files and compared each regex grader against the added comment text. None of the patterns can match the marker's words, and none depends on the exact end of the draft. Whether or not a rewrite echoes the marker, a grader result can't change because of it.
  • Residual, below the blocking bar: the eval prompts now carry about 12 extra words. The PR body's measured scores (1.000 on all cases) came from runs of the prior prompt text, and I did not re-run them. The comment is outside every grader's reach, so I'd expect no effect, but I haven't confirmed that. The be-concise skill may count the marker's words in its "before" word count, which affects only the reported number and not any grader.

Out of scope for this incremental pass: all other files in the PR were reviewed at efc9497 and are unchanged.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 23a258658d3e10bb994d80504a7d69ad498c6d2f: 0 finding(s), 0 as inline comments.

Security review (incremental, HEAD 23a2586)

No security findings.

Scope: the change since efc9497, in plugins/writing/evals/bc-pr-body-floor/prompt.md and plugins/writing/evals/bc-status-point-first/prompt.md.

  • Data only: Each file gains one `` HTML comment at the end of a line. The diff adds no code, command, workflow, permission or secret handling, and it touches no trust boundary.
  • No standing instruction weakened: The diff deletes, narrows or softens no rule in CLAUDE.md, AGENTS.md, a rules file, a skill or agent body, or a hook allowlist. The marker only exempts these two deliberately wordy eval drafts from the ai-slop lint. That exemption sits on eval fixtures under plugins/writing/evals/, and I found nothing in the diff that carries it to other files.
  • No injection risk: The comment text is a fixed descriptive string with no instruction aimed at the model. The eval feeds these drafts to be-concise only as text to shorten.
  • Protected-class list not consulted: I applied the instruction-surface lens by consequence and did not read docs/conventions/instruction-exception-register/README.md. Nothing in the diff needed a protected class to decide it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 20, 2026
Merged via the queue into main with commit 58b75f3 Oct 20, 2026
25 checks passed
@kyle-sexton
kyle-sexton deleted the feat/concise-skills-writing branch October 20, 2026 05:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concise skills rollout: writing (1 skills, 1 model-invoked)

1 participant