[kyle-sexton]

Closes #6611

Summary

Shortens the naming:name-it-better description from 650 to 624 characters on the description-only path. The body is unchanged. The new description scored the same as or better than the old one on held-out trigger probes and on four routing cases, on both Opus 5.5 and Sonnet 5.5.

Fix

• plugins/naming/skills/name-it-better/SKILL.md line 2 is the only changed line (git diff origin/main -- plugins/naming/skills changes nothing outside the description key). A fresh rewriter worked on a copy outside the repository. It could not read the eval cases, the probes, the issue or the git history. It made three edits:
  1. "Then let the human pick." and "; never auto-locks a name" are merged into one sentence: "The human picks; never auto-locks a name."
  2. "UNDECIDED, whether the ask is" becomes "UNDECIDED:".
  3. The two decided-rename sentences become one, ending ", which routes to the rename-references sweep".
• Kept word for word: every quoted trigger phrase, the list of target kinds, both UNDECIDED branches, the decided-rename exclusion and where it routes, the blind-generators clause and the tournament argument.
• .changes/naming/ patch fragment.
• The frozen probe file plugins/naming/probes/name-it-better.json landed on main in test(naming): add outcome eval cases and frozen trigger probes for name-it-better #12002 and is present on this head unchanged.

Verification

Runs used Claude Code 2.1.316 with before and after copies launched together. Each pass used --ablation none and the same flags. Every number below comes from a pass that run-validity.py judged VALID (origin/main copy, with --scaffolded for the scaffolded cases). The decision rule was preregistered before the first paid run, and the eval suite was frozen at the digest recorded on #6611 (plugins/naming/evals tree 1afdcd49).

Trigger scores (/skill-quality:check measure-invocation). 16 held-out probes were written blind by a skills-disabled print session from a neutral statement of the skill's purpose. They were emitted with measure-invocation.sh emit-plugin-eval --runs 3.

Model	Should-trigger fired, before	Should-trigger fired, after	Should-not quiet, before	Should-not quiet, after
Opus 5.5	24/24	24/24	24/24	24/24
Sonnet 5.5	23/24	24/24	24/24	24/24

• No trigger was lost.

• The noise report calls the after version non-inferior at a 0.05 margin on both models.

• On the frozen probe file, the lexical measure-invocation.sh score gives:

  • before: train 0.8 trigger / 0.2 false, validation 0.6 / 0.4
  • after: train 0.8 / 0.2, validation 0.4 / 0.4

  This score is a tripwire, not the verdict. The one validation probe that flipped is pos-val-04 ("Help me pick a filename for the doc that explains how releases get cut"). Its overlap score fell from 9 to 8 when "Then let the human pick" became "The human picks", because the probe word "pick" no longer matches exactly. At 8 it ties with docs-naming:generate-file-name-gate, so the lexical score no longer predicts a trigger. The skill criteria say "A deterministic lexical score is a tripwire only, never the verdict" (Measurement, step 6). The Done-when comparison is the preregistered model-graded held-out score in the table above, and it did not drop on either model.

Routing cases. 5 runs per version, with-arm mean scores:

Case	Opus before	Opus after	Sonnet before	Sonnet after
nib-decided-rename-not-routed (must not route)	1.000	1.000	1.000	1.000
nib-rejected-name-natural-phrasing	1.000	1.000	1.000	1.000
nib-blast-radius-basis	0.867	0.867	0.800	0.933
nib-misleading-name-no-apply (Bash/Edit/Write granted)	1.000	1.000	1.000	1.000

• In every pass, the routing grader gave the same result for both versions: 5 of 5 routed on the three should-route cases, and 0 routes on the decided-rename case.
• No must-not grader failed.
• All blast-radius-basis misses are the basis-label grader. Every missed reply was read. Each one states the searched reference count, and most carry a Basis line in a form the pattern does not match (for example "Basis for chore: initialize marketplace scaffold #1: the blast radius is verified"). This grader-recall limit is noted for a later round.
• nib-tournament-public-package was not run: it is slash-invoked, so the description cannot affect it, and the body is unchanged.

Meaning diff. Two /docs-hygiene:compress compare labelers both returned PASS, with 0 SEMANTIC LOSS and 0 AMBIGUITY.

/skill-quality:check plugins/naming. PASS on main (0 errors, 3 warnings) and PASS on this head (0 errors, 4 warnings). No new FAIL.

• The added warning says the description dropped the trigger keyword "the target name is still UNDECIDED". That phrase is still in the description word for word; only the punctuation after it changed (a comma became a colon). This is a check false positive.

Cost. 28.53 USD list-price estimate for 21 passes (host-safety probe included). Every pass stayed under the 5 USD per-pass ceiling.

Related

• Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476 (Concise skills rollout parent)
• test(naming): add outcome eval cases and frozen trigger probes for name-it-better #12002 (frozen outcome cases and trigger probes)

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-02T00:33:16.101467Z	0be063b	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed 0be063bc9eb4533f310d7dad124b2e6d207eddc5: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review: no security issues found

I read the whole diff for both changed files. Neither file matches a trusted-path glob.

• plugins/naming/skills/name-it-better/SKILL.md: only the frontmatter description value changed. argument-hint, user-invocable and disable-model-invocation are the same, and so is the skill body. The change shortens a standing instruction, so I read it under the instruction-surface lens before applying any skip. It removes no rule:
  • "never auto-locks a name" is still there.
  • The human-picks rule is still there. "Then let the human pick." became "The human picks".
  • The exclusion for a rename that is already decided, and where that request routes, is still there.
  • The change grants no tool, adds no hook and adds no allowlist entry.
  • Nothing that could reach a lane holding credentials is added.
• .changes/naming/feat-6611-concise-naming-3e7a1c52.md: a new changelog fragment that is plain prose with a bump: patch frontmatter key. It has no security surface.

Other checks:

• Manifests: no plugin or marketplace manifest changed.
• Trust controls: no trust control, token scope, kill switch or merge hold changed.
• PR context: the description and the closing issue's comments contain no text telling this lane what to do.
• Spend: the PR body reports about 28.53 USD (list-price estimate) of eval runs already done on the host. That is usage the author already spent, not a charge this change creates, so it is not a finding and not a hold.
• Prior findings and cap: this lane has no prior findings on the PR, and the round cap (0 of 3 rounds used) does not apply.

critical-findings: 0

[github-actions]

claude-review reviewed 0be063bc9eb4533f310d7dad124b2e6d207eddc5: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Code review: no findings

I read both changed files in full: plugins/naming/skills/name-it-better/SKILL.md and .changes/naming/feat-6611-concise-naming-3e7a1c52.md.

What I checked

• The description edit. Only the description key changes. I compared the old and new text clause by clause:
  • Every quoted trigger phrase is still there, word for word.
  • The list of target kinds, both UNDECIDED branches and the decided-rename exclusion are kept. So are its route to the rename-references sweep, the blind-generators clause and the tournament argument.
  • "never auto-locks a name" moved next to "The human picks". The meaning is unchanged.
• The length claim. I counted the four edits by hand: +17, −19, +1 and −25 characters, a net −26. That matches the stated 650 → 624.
• The changelog fragment. It uses the same frontmatter and ### Changed layout as the other .changes/* fragments. bump: patch fits a change to description wording only.
• Path-scoped rules. skill-bodies-state-current-rules.md and cost-claims.md both match the changed SKILL.md. The new description restates no volatile upstream detail and no cost, so neither rule is broken.

Closing issue #6611: done-when check

• Only the description key changes under plugins/naming/skills: met.
• plugins/naming/probes/name-it-better.json exists on this head. It landed earlier in test(naming): add outcome eval cases and frozen trigger probes for name-it-better #12002 and this PR doesn't change it: met.
• The PR body shows before and after trigger scores from measure-invocation on the same probes: met.
• After scores at least as well as before: met on the model-graded held-out scores, per the author. The lexical tripwire score did drop on validation (trigger 0.6 → 0.4). The PR body explains why: the probe pos-val-04 lost its exact match on "pick". It also cites the skill criteria, which call the lexical score a tripwire only, never the verdict. I treat this as addressed, not as a finding.
• /skill-quality:check plugins/naming reports no new FAIL: the author claims 0 errors on both main and this head.
• A changelog fragment is present: met.

Every eval score, the /skill-quality:check results and the meaning-diff PASS come from the author. I could not run any of them in this lane, so they are unverified.

Not a hold: the change itself spends nothing. The author's eval passes drew on prepaid API credit (an estimated 28.53 USD at list price, as stated in the PR).

No manifest files changed. There are no prior findings or settled threads, and the round cap does not apply. I found no embedded instructions in the PR context.