guardrails: rescope asserted-path-verify — 23.7% firing rate, zero true positives on a full-corpus sweep

[kyle-sexton]

Observation

An asserted-path-verify guard was built under #1270 and withdrawn from #1284 on measurement, not on judgment. This issue carries the measurement so the idea can be rescoped rather than rediscovered.

The guard flagged a repo-relative path asserted in markdown — in a code span or a link destination — that does not exist in the working tree. Oracle: a filesystem test. It shipped a "first-segment gate" intended to keep it quiet: a candidate was only adjudicated when its leading directory existed in the repo.

What the sweep found

Every tracked markdown file in melodic-software/claude-code-plugins (975 of 975), each fed to the hook as a real PostToolUse Write payload carrying that file's actual content and path.

Files firing	231 / 975 = 23.7% — roughly one in four markdown writes
Findings	389 total, 145 distinct
True positives	0

Not concentrated in low-traffic corners: plugins/** 22.7%, docs/topics/** 24.6%, docs/conventions/** 45.8%. Eight plugin SKILL.md and eight plugin README.md files fired — the highest-traffic docs in the fleet. Worst single file: docs/conventions/config-cascade/README.md, 13 findings.

The oracle never misfired. No candidate resolved at the repo root. Every finding was a scoping problem.

False positives by cause

n	%	Cause
281	72.2%	Consumer-project config surface. .claude/** (240), docs/{standards,specs,decisions,adr} (~25), .claude-plugin/** (10), .github/** (4). These document a consuming repo's tree — e.g. plugins/testing/skills/run-e2e/SKILL.md cites .claude/testing/e2e.md, which a consumer creates and this repo correctly lacks. The first-segment gate passed only because a marketplace happens to carry same-named top-level directories.
65	16.7%	Subtree-relative citation. lib/**, scripts/** cited relative to the citing doc's skill root or plugin root, and present there. E.g. plugins/ai-briefing/skills/generate/references/build-pipeline.md cites lib/emit-slides.js, which lives under that skill's own output/build/lib/.
35	9.0%	Forward-looking. PLAN items marked CREATE, NOT_IMPLEMENTED.md docs describing what would exist, scripts named in CHANGELOGs as retired. The guard's own header documented this as an accepted residual.
6	1.5%	Illustrative placeholders — docs/example.md, docs/foo.md. Two sit in plugins/claude-memory/skills/audit/reference/criteria.md, a doc about detecting broken references.
1	0.3%	Link destination nested inside a code span. plugins/code-tidying/skills/tidy/reference/tidyings.md:151 writes a markdown link as literal example text inside backticks; the code-span and link extractors ran independently over raw content, so the destination was extracted anyway.
1	0.3%	Historical record — a retired script named in a completion record.

Why this is a rescope, not a tuning pass

Fixing causes 1, 2 and 5 removes 347 of 389 (89%). The residual is still ~4% of files firing with zero true positives. A repo-root filesystem test is the wrong oracle for a repo whose docs are largely about other repos' trees.

The one place it would earn its keep is instructive: docs/topics/fresh-eyes-checkpoint-audit/PLAN.md:200 cites plugins/re-anchor/context/re-anchor-audit-correct.md, which #1276 moved to plugins/discipline/context/. Verified genuine — and it exists only on main, so the guard found nothing real on its own branch.

Direction (open, deliberately)

The measurement covered one repo, and it is the pathological case for this guard. In an ordinary application repo, .claude/** paths do exist locally and a doc describes its own tree — so the guard may be mis-scoped rather than unsound. Candidate shapes, none chosen:

• Scope by repo class. Inert in a marketplace repo (one with plugins/*/.claude-plugin/plugin.json), active elsewhere. Inverts skill-reference-verify's gate, which is only meaningful inside a marketplace.
• Subtree-aware bases. For a code span in a file under plugins/<p>/, also try plugins/<p>/skills/<s>/ and plugins/<p>/ — removes cause 2 outright.
• Exclude consumer-convention roots — never adjudicate a candidate rooted at .claude/, .claude-plugin/, .github/, or a documented consumer docs root. Removes cause 1, the 72%.
• Strip code-span regions before the link extractor — removes cause 5. Unambiguous and cheap regardless of the rest.

Whatever shape is chosen must be re-measured against the same corpus before shipping. The 23.7% figure is the bar to beat, and "it looks better now" is not a measurement.

Acceptance

• A firing rate re-measured across all tracked markdown, reported as a number
• Either a true positive on the real corpus, or — if the corpus contains no violation
  of the class — detection established by seeding known defects, with the unseeded
  firing rate reported as the noise figure. A silent sweep leaves precision undefined,
  not zero; only firing with no true positive disqualifies
• The enforceability tier stated honestly for whatever oracle is chosen
• Repo-agnostic: no behavior that only makes sense in this marketplace

Related

Withdrawn from #1284 (which shipped its sibling skill-reference-verify). Scoped originally in #1270. The re-anchor → discipline rename that produced the one genuine finding is #1276.

---

This was generated by AI during a session on the fact-check capability.

[kyle-sexton]

This was generated by AI during admission.

Admission read. status: needs-decision was already applied but the needs-human role label
was not, so this kept re-surfacing on list-frontier --autonomous as if it were dispatchable. It
isn't: the body carries an explicit "Trigger: revisit when …" deferral and/or a maintainer-owned
design fork with no default this lane can pick — that is a genuinely open decision, not a briefed
fix. Applying needs-human to close the labeling gap; no change to the substance of the deferral.