docs-hygiene: audit-derivability has no verdict for a routing-only agent doc, and keep-sample spot-tests have no defined outcome

[kyle-sexton]

Summary

A repo-sweep run of /docs-hygiene:audit-derivability sweep . (0.23.1) on melodic-software/.github could not give CLAUDE.md a verdict. That file is 41 lines, agent-facing, and only routes to README.md, the ci.yml header, and .claude/rules/pr-body-contract.md. Commit e9d8343 created it on purpose. The operator had to invent a "provisional" label and ask the user. The installed SKILL.md and context/rubric.md match origin/main, so all four findings still apply.

Findings

F1 (medium): a routing-only agent doc has no verdict path.

• The Factor 1 table in rubric.md (lines 44-49) has no row for a "fact X lives at Y" claim. Such a claim neither restates code nor copies another doc.
• The fresh agent's converged spot-test drew its answers from markdown (README.md, pr-body-contract.md). The rubric (lines 25-29) excludes prose docs as derivation sources, so "converged" did not show derivability as the rubric defines it.
• The convert-to-pointer fix, "replace the body with a one-line pointer" (SKILL.md line 56), does nothing for a doc that is already pointers.
• A root CLAUDE.md loads at launch, and it is exactly what tells an agent where to look. So "an agent could just look" (rubric.md 133-134) does not apply to it.

F2 (medium): the keep-sample spot-test has no defined consequence. SKILL.md line 85 requires spot-testing a sample of keep verdicts. The protocol (rubric.md 158-163) defines outcomes only for delete/pointer verdicts. Nothing covers what a converged keep-sample triggers, how big the sample is, or how it is chosen. The output schema and aggregate have no field for it. context/derivability-route-followups.md lines 125-129 require recording the sample set, but SKILL.md does not say so.

F3 (low-medium): the git-log check for deliberate state only covers empty or near-empty files (SKILL.md line 74, rubric.md line 179). A deliberately created doc whose commit records a decision is not covered. The rubric also contradicts itself: line 47 grades git history as derivable, while the line 179 example treats a decision recorded in a commit as an owned fact.

F4 (low, not yet hit): the spot-test agent can already have seen the doc. The protocol says "e.g. an Explore agent". Per the sub-agents docs, only Explore and Plan skip CLAUDE.md, and every other subagent type loads CLAUDE.md and project rules. A spot-test of those files on any other agent type is graded by an agent that has already read them.

Fix

Keep every change in the rubric body and the evals, not the frontmatter description. #4142 records the listing budget at 7986/8000 characters.

1. F1: add a Factor 1 row plus a worked example for routing claims. The outcome is convert-to-pointer (already satisfied), not actionable, and counted in the aggregate. Add one sentence to spot-test step 2: for a router, do not hand the fresh agent the trigger as a question.
2. F2: add a protocol paragraph: a diverged keep-sample confirms the keep; a converged one sends the doc back through the four factors, and any actionable result ships as provisional. Add "sampled/overturned" counts to the aggregate, and move the recording rule into SKILL.md.
3. F3: make the git-log check a precondition for every delete/convert-to-pointer verdict. When a commit records a decision about the doc, the verdict ships provisional with "reverses a recorded decision". Update the line 179 example.
4. F4: require Explore or Plan when the audited file is loaded at launch.
5. Add evals: a pure-routing CLAUDE.md, a converged keep-sample, and a deliberately created doc.

Verification

• skill-quality:check: PASS, 1 WARN. The WARN is check 21 at rubric.md:147, the spot-test protocol that itself prescribes fresh-context delegation, so it is a false positive.
• Every doc citation was quote-checked against pages fetched 2026-09-27.
• Evals not run.
• Not checked: whether the sibling audit-progressive-disclosure has the same routing-doc gap.

Related

• docs-hygiene: agree the plugin's contract before it grows again #4142 (plugin charter and listing budget)
• Evidence session: melodic-software/.github sweep PR chore(skill-quality): harden static checker + record terminal retrofit scope #153

[kyle-sexton]

Reopened. This comes from the 2026-09-29 audit of the Cursor run.

What the audit found. #4868 closed this issue COMPLETED but delivered F1 only, and it used a closing keyword. Even F1 was partial: the new row in context/rubric.md contradicted the rubric's own definition of "derivable" (lines 25-29). F2, F3, F4 and the evals were never delivered.

Fixed on fix/audit-docs-hygiene, PR #5296 (draft). Paths are under plugins/docs-hygiene/skills/audit-derivability/.

• F1: the routing-index row in context/rubric.md is now a named exception to the derivable-from-primary-sources rule and never yields delete for a launch-loaded routing doc.
• F2: a new "Keep-sample protocol" section in rubric.md covers the size rule, the diverged and converged outcomes, and the recording rule. SKILL.md gains sampled and overturned counts in the aggregate line and output schema.
• F3: git log runs before every delete and convert-to-pointer. A verdict on a doc whose commit records a decision ships provisional as reverses a recorded decision.
• F4: a launch-loaded audited file must be spot-tested by Explore or Plan, with a four-part verification record against the sub-agents doc.
• Evals: 14, 15 and 16 plus one fixture (routing-only root CLAUDE.md, converged keep-sample, deliberately created doc).

The PR uses Closes #4573 only because all five items are met on the branch. The issue stays open until you merge #5296. check-evals-quality.sh passes on all 13 evals.json; the evals have not been run against a model.

[kyle-sexton]

Closed by #5296 (merge commit 3c0b856). All five items in this issue are met on the merged branch. The squash commit did not carry the Closes #4573 keyword because the PR body was overwritten by a scratch-file collision between parallel babysitting sessions during the merge; the body is restored, so this issue is closed by hand.