Repository navigation
claude-ops: audit-skill-visibility emits per-skill starvation verdicts when every score is zero #4077
Copy link
Copy link
Closed
Labels
needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.
Description
Summary
When no competing skill has any recorded usage (every fresh install, every cloud container), the product's stable sort keeps catalog order and the first-fit walk grants the first entries that happen to fit. audit-skill-visibility models this correctly (score_basis: "unscored", per-row confidence: "unscored") but still emits verdict: "likely-starved" and a numeric band on every competing row, and its Markdown says "roughly N of them are running name-only". Readers take that as a usage-derived ranking. It is not: in the reproducing run the "starved" list was simply the catalog from the letter C onward.
Evidence
- Binary (Claude Code 2.1.263): both truncators sort descending by
usageCount * max(0.5^(days/7), 0.1)with a stable sort, then walk first-fit with no break. At all-zero scores the ordering is catalog position. reference/listing-scorer.mdalready states this ("When the band means nothing") and the classifier header says "A confidently wrong band is worse than no band", yetcompute_listing(around lines 896-937) still writes the verdict and band, and the renderer (around lines 1316-1324) still prints the count-as-ranking sentence.- SKILL.md's refusal doctrine: "routes any claim the span cannot support into a first-class
withheldsection with its reason. A declined verdict is reported, never omitted." - The
withheldlist built inbuild_model(around line 1115) only carries observation claims today.
Expected behaviour
- Keep the arithmetic: demand, budget, overflow, and the count of entries that cannot fit stay in both renders.
- WHILE
score_basisisunscored, no competing row carriesverdict: likely-starvedor aband; each such row's starvation object readsverdict: withheldwithreason: unscored. - The per-skill "which ones" claim moves into the
withheldsection with reason "unscored: ordering is catalog-order tie, not usage", once, not once per skill. - The Markdown drops "roughly N of them are running name-only" in the unscored case and prints the cannot-fit count as a count.
Acceptance criteria
- [state-driven] WHILE every competing usage score is 0, the JSON contains zero rows with
verdict: likely-starvedand onewithheldentry with reasonunscored. - [event-driven] WHEN at least one competing skill has non-zero usage, per-row verdicts and bands render exactly as today.
- [ubiquitous]
starved_countandoverflow_charsare unchanged in both cases. - [ubiquitous]
OverflowConsumptionTestandListingScoreTest.test_all_zero_scores_report_an_unscored_basisare extended to assert the withheld shape. scripts/affected-tests.sh --runis green.
Out of scope
- The budget inputs themselves (settings fraction, window, bytes-per-token): claude-ops: audit-skill-visibility derives the listing budget from hardcoded defaults instead of the effective settings and model #4076.
Related
- Sibling: claude-ops: audit-skill-visibility derives the listing budget from hardcoded defaults instead of the effective settings and model #4076 (settings-derived budget inputs); the enabledPlugins reachability issue is linked in the first comment.
- Binary re-verification note: the tiebreak is catalog order via stable sort;
listing-scorer.mdsays "alphabetical" in one place and "catalog order" in another, and should say the latter in both.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.
Activity
kyle-sexton commentedon Sep 11, 2026
Sibling issues from the same audit run: #4076 (settings-derived budget inputs) and #4078 (enabledPlugins reachability).
Generated by Claude Code
fix(claude-ops): withhold per-skill starvation verdicts when every us…
feat(claude-ops): audit-skill-visibility names misconfigured skills, …