skill-quality: check-skill.sh has no shared listing-budget check, plus three smaller gate defects

[kyle-sexton]

Why this exists

The skill-listing measurement behind #1271 (evidence PR #1286, closed — its diff stays readable)
audited what skill-quality:check already enforces before proposing anything new. It found four
defects in the incumbent gate. They are recorded here because #1271 is being re-scoped to a decision
about which levers to pull, and these are code fixes that stand on their own regardless of how that
decision lands.

1. There is no check on the shared listing budget at all — the important one

plugins/skill-quality/scripts/check-skill.sh:219-234 (check 2) is a per-skill cap: it guards
the 1,536-character per-entry truncation limit. The skill's own description advertises a
listing-budget cap, which is a different limit — the shared budget across every listed skill,
documented as scaling with 1% of the model's context window.

Nothing checks that shared budget. That is how the listing came to overflow by a large multiple
with no gate reporting it. The measured per-entry maximum in this tree is 1,161 characters against
a 1,536 cap, so check 2 has never fired and never would; meanwhile the aggregate is multiples over.

A check here has to decide what to do about a machine-dependent budget (it scales with context
window, and the drop order is decayed local usage), so the honest form is probably a reported
aggregate with a repo-level ceiling rather than a pass/fail on live drop behavior.

2. Check 2 omits the entry joiner

check-skill.sh:227 computes DESC_LEN + WTU_LEN. The actual entry text the harness builds is
description + " - " + when_to_use — a literal three-character joiner, recovered from the
shipped binary rather than inferred. The check under-counts by 3 whenever when_to_use is populated.

Not currently binding, and it will not become binding at present lengths. It is wrong in exactly the
direction the listing work is about, and it is a two-character fix.

3. Check 12 penalizes the authoring model when_to_use exists to enable

check-skill.sh:440-448 warns unless the literal text use when appears in description or
when_to_use. A skill that puts bare quoted trigger phrases in when_to_use with no prose marker
warns — measured at 15 of 15 on a prepared rewrite block.

Two fixes, and they are not equivalent:

• Keep a literal Use when: prefix inside when_to_use. No code change, but it costs roughly ten
  characters per skill on top of the joiner's three — across 130 model-invocable skills, about
  +1,700 characters, which is more than the entire recoverable mass the measurement found.
• Amend check 12 to accept a populated, single-quoted when_to_use as satisfying the
  trigger-spec requirement. This is the better fix: the warning text ("a description is a trigger
  spec, not a summary") encodes the pre-when_to_use authoring model.

The second edits check-skill.sh, so it sequences behind #1096, which is already editing that
file and claims check 21.

4. skillListingBudgetFraction is not at its documented default on at least one machine

The measurement found the live value is 0.03, not the documented 0.01. Any proposal that reasons
from the default is reasoning from the wrong number on that machine. This is a consumer setting
rather than a repo artifact, so it belongs here as a caveat on any check built for item 1: a gate
that assumes 0.01 will disagree with /doctor on a machine that sets it otherwise.

One discrepancy worth resolving before building item 1

Two figures for the same budget are on #1271 and they differ by 4×:

• Derived from the shipped binary: the budget is contextTokens × 4 × 0.01 characters — 40,000
  at 1M context, 8,000 at 200k.
• Read directly from the docs' "1% of the model's context window": 10,000 at 1M, 2,000 at 200k.

The difference is whether the 1% applies to tokens (then multiplied to characters) or to characters
directly. It changes the reported overflow multiple from roughly 2× to roughly 8× at 1M context.
Settle this against the binary or against /doctor's reported listing cost before any check
asserts a number, since /doctor already reports the listing's context cost and its biggest
contributors, and overflow writes a warning to the debug log under --debug.

Related

Refs #1271 (the listing-budget issue these were found under, now status: needs-decision).
Refs #1096 (owns check-skill.sh today; item 3 sequences behind it).
Refs #1286 (closed evidence PR — the full measurement, per-plugin tables, and re-verification record
live in its diff).

[kyle-sexton]

This was generated by AI during triage/admission.

Admission read. No work-class: label — unclassified, and per this lane's admission gate that
fails closed to human-gated regardless of how mechanical any individual part looks. Applying
needs-human; not this lane's call to set a work-class.

Separate from the classification gap, the item itself bundles at least four defects of visibly
different shape in one issue: one ("no shared-listing-budget check") reads as needing a design
decision about what a machine-dependent budget check should report; another (the entry-joiner
undercount) reads as a small, mechanical, self-contained fix; a third is explicitly sequenced behind
#1096. A human unbundling this into per-defect tickets (or ratifying the bundle as one C3 item) would
let at least the mechanical piece move independently of the design and sequencing questions in the
others.

[kyle-sexton]

This was generated by AI, delegated directly by the repository operator to resolve this blocker.

Agent Brief

Type: C3 scoped bug fix (bundle ratified as one item — see Scoping decision below).

Summary: Fix three verified defects in the skill-quality static gate: add a repo-level
aggregate listing-mass check (new sibling script, ratchet ceiling), correct check 2's 3-character
joiner undercount, and amend check 12 to accept a populated single-quoted when_to_use as a valid
trigger spec. Item 4 (machine-varying skillListingBudgetFraction) is honored as a design
constraint, not a code change.

Verification against current origin/main (this session)

All line references re-verified against origin/main:plugins/skill-quality/scripts/check-skill.sh
(741 lines, 20 checks):

• Defect 1 confirmed. No aggregate/shared-budget check exists anywhere in the script — the only
  budget mentions are header comments (lines 64–66). Check 2 (lines 219–234) is strictly
  per-entry against DESC_CHAR_CAP=1536 (line 132).
• Defect 2 confirmed. Line 227: COMBINED_LEN=$((DESC_LEN + WTU_LEN)) — omits the harness's
  3-character " - " joiner between description and when_to_use. (The joiner itself was
  recovered from the shipped binary in docs(topics): commit the skill-listing budget measurement for #1271 #1286's measurement; this session could not independently
  re-verify the binary — implementer re-verifies per the fresh-docs mandate before relying on it.)
• Defect 3 confirmed. Lines 440–448: check 12 warns whenever the literal text use when is
  absent from description+when_to_use, even when when_to_use carries single-quoted trigger
  phrases that skill_frontmatter::extract_triggers would recognize.
• feat(skill-quality): fresh-eyes delegation doctrine + conformance gate (check 21) #1096 is OPEN (needs-human), still claims check 21, has not landed.

Current behavior

1. Nothing measures the aggregate listing mass across skills; the listing overflowed by a large
   multiple with no gate output (per docs(topics): commit the skill-listing budget measurement for #1271 #1286's measurement — per-entry max 1,161 < 1,536, so check 2
   has never fired).
2. Check 2 under-counts every entry with a populated when_to_use by 3 characters.
3. Check 12 warns on the exact authoring shape when_to_use exists to enable (bare quoted trigger
   phrases, no Use when: prose marker).

Desired behavior

1. New repo-level script plugins/skill-quality/scripts/check-listing-budget.sh (plus a
   check-listing-budget.test.sh, matching the existing scripts/*.test.sh convention that check 7
   already enforces):
   • Computes aggregate listing mass over every model-invocable skill under the resolved skills
     root: len(description) + (when_to_use populated ? 3 + len(when_to_use) : 0) per entry,
     summed. Skills excluded from the model listing are excluded from the sum; the implementer
     verifies the exact opt-out frontmatter key against current official docs (fresh-docs mandate —
     this is documented harness surface).
   • Always reports the aggregate (note line). FAILs only above a repo-level ratchet ceiling: a
     constant in the script, initialized to the measured aggregate at implementation time rounded up
     to the next 1,000 characters. The gate is a no-net-growth ratchet, deliberately NOT a model
     of the live machine budget: skillListingBudgetFraction varies per machine (0.03 observed vs
     0.01 documented — item 4), so a live-budget assertion would disagree with /doctor somewhere.
     One header comment records that constraint.
   • Wire it into the check action in skills/check/SKILL.md as a once-per-invocation repo-level
     step (not per-skill — running an aggregate sweep inside per-skill check-skill.sh would repeat
     identical output N times).
2. Check 2 joiner: line 227 adds 3 when WTU_LEN > 0.
3. Check 12: passes when when_to_use is populated AND
   skill_frontmatter::extract_triggers over it is non-empty, even without literal use when
   text. The existing second warn branch (unquoted triggers) is retained. Warning text updated so it
   no longer encodes the pre-when_to_use authoring model.

Scoping decisions (binding, per operator delegation)

• Remedy for defect 3: amend check 12 (the issue's second option). The alternative — a literal
  Use when: prefix in every when_to_use — costs ~+1,700 characters fleet-wide, more than the
  entire recoverable mass docs(topics): commit the skill-listing budget measurement for #1271 #1286 found, actively worsening the budget defect 1 gates. Rejected.
• Ceiling semantics for defect 1: ratchet, not live-budget model. This makes the unresolved
  4× formula discrepancy on skill metadata: trigger phrases crammed into description; when_to_use is unused and the shared listing budget silently drops them #1271 (tokens×4×1% vs 1% of chars) non-blocking here: it only
  matters when choosing how far to tighten the ceiling below the ratchet, which is exactly skill metadata: trigger phrases crammed into description; when_to_use is unused and the shared listing budget silently drops them #1271's
  lever decision. Deferred there with that trigger.
• Sequencing vs feat(skill-quality): fresh-eyes delegation doctrine + conformance gate (check 21) #1096: soft coordination, not a hard block. feat(skill-quality): fresh-eyes delegation doctrine + conformance gate (check 21) #1096 is stalled at needs-human;
  hard-blocking would strand a two-line fix indefinitely. The aggregate check lands as a sibling
  script — deliberately NOT claiming check 21, which feat(skill-quality): fresh-eyes delegation doctrine + conformance gate (check 21) #1096 owns — and the check-12 edit touches
  lines 440–448, disjoint from an appended check 21. Whichever lands second rebases; conflict
  surface is limited to the "twenty checks" count prose, which this issue does not change (the
  per-skill count stays 20).
• Bundle ratified as one C3 item (the triage comment's alternative to unbundling): items 2–3 are
  two-line edits in the file item 1's work already touches the plugin around; four tickets would
  cost more coordination than the work itself. Item 4 requires no code — it is honored by the
  ratchet design.

Acceptance criteria

• check-listing-budget.sh exists, reports the aggregate on every run, exits nonzero only above
  the ratchet ceiling, and passes on current main.
• check-listing-budget.test.sh covers: under-ceiling pass, over-ceiling fail, joiner counted
  only when when_to_use is populated, opt-out skills excluded.
• check-skill.sh:227 counts the 3-char joiner when when_to_use is populated;
  check-skill.test.sh proves it.
• Check 12 no longer warns on a populated, single-quoted when_to_use without literal
  use when text; still warns when both description lacks the phrasing and when_to_use is
  empty or unquoted; check-skill.test.sh proves both sides.
• skills/check/SKILL.md documents the repo-level step; per-skill check count prose unchanged.
• Harness-behavior claims (the " - " joiner, the listing opt-out key) re-verified against
  current official docs / shipped binary per the repo's fresh-docs mandate, cited in the PR.
• plugins/skill-quality/.claude-plugin/plugin.json version bumped; CHANGELOG entry added.

Out of scope

• Choosing listing-reduction levers or tightening the ceiling below the ratchet — skill metadata: trigger phrases crammed into description; when_to_use is unused and the shared listing budget silently drops them #1271.
• Check 21 / fresh-eyes conformance — feat(skill-quality): fresh-eyes delegation doctrine + conformance gate (check 21) #1096.
• Settling the 4× budget-formula discrepancy — recorded on skill metadata: trigger phrases crammed into description; when_to_use is unused and the shared listing budget silently drops them #1271; non-blocking under the ratchet
  design.
• Any change to consumer machines' skillListingBudgetFraction.

[kyle-sexton]

[kyle-sexton]

Batch-4 filing note (this issue is closed; recorded because its engine is the prerequisite for a newly-filed row).

#2274 files claude-config:audit Category G, whose B3 row is not "write a measurement script" — the originating item proposed exactly that, and check-listing-budget.sh from this issue already exists. The correction the fix needs: the two tools measure different populations. This script walks this repository's authored skills roots (a marketplace publisher gate). Category G audits a consumer machine's installed roster, which needs a plugin-cache walk, version resolution across version directories (sort -V, not lexical — twelve plugins in the cache disagree, claude-config itself among them), and enabledPlugins reading that this script has no step for.

So Category G cannot simply call it. The honest options are (a) point G at it as an in-repo proxy and say so, or (b) extend it with a cache-walk mode.

Also relevant to anyone extending it: this script uses no jq and its frontmatter fence is CR-tolerant, so two of the four platform traps the originating item listed are already neutralized by construction; the process-spawn cost is the one that still applies, and it is tracked at #2216.