From 9f7e2fb23e602a7efe0d63477b0e35b76424b68f Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 25 Jul 2026 12:27:01 -0400 Subject: [PATCH 1/5] docs(adr): record that verification guards earn default-on by measured precision MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #1352 The #1270 program scoped three claim-verification guards on sound oracles and shipped one. The generalizable finding — a sound oracle is necessary but not sufficient — lived only in closed PRs and issue bodies. The asserted-path guard reached 61 green contract cases and a full review round before a corpus sweep showed 23.7% of files firing across 389 findings with zero true positives. Its oracle was exact; its scope was a repo whose docs mostly describe other repos' trees. A passing contract suite proves the oracle; only a corpus sweep proves the scoping. Records four rules: measure against the real corpus before shipping default-on, report the number in the PR, treat zero true positives as disqualifying however sound the oracle, and distinguish wrong-oracle from wrong-scope because that decides deletion versus rescoping. Also records the cost — measuring right after the guard first worked, instead of after polish and review, would have saved that review round. Extends ADR 0002's verification-promotion discipline one step earlier in the lifecycle: 0002 governs promoting an advisory gate to blocking on demonstrated precision, this governs whether it ships default-on at all, on the same evidentiary basis. Defers to docs/conventions/hook-precision/README.md for the post-ship over-fire discipline rather than restating it. Co-authored-by: Claude Opus 5 (1M context) --- ...s-earn-default-on-by-measured-precision.md | 99 +++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md diff --git a/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md b/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md new file mode 100644 index 0000000000..989256f4c1 --- /dev/null +++ b/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md @@ -0,0 +1,99 @@ +# Verification guards earn default-on by measured precision, not by plausible oracle + +- Status: accepted +- Date: 2026-07-25 + +## Context + +A program to add claim-verification guards to `guardrails` (#1270) scoped three, on the +reasoning that each had a mechanical oracle and no existing coverage: + +| Candidate | Oracle | Tier claimed at scoping | +|---|---|---| +| Asserted repo-relative path that does not exist | filesystem test | Deterministic | +| Version string disagreeing with its owning manifest | manifest compare | Deterministic | +| `/plugin:skill` reference that does not resolve | glob the plugins tree | Detect-then-judge | + +All three reasoned soundly from the oracle. Two did not survive contact with the repository. + +The **version guard** was dropped before implementation. Its only in-repo surface — a +`## []` heading versus the manifest — is already covered deterministically by +`scripts/check-changelog-parity.sh --check-bump`, a required CI gate. An enumeration of the +residual prose surface (all `*.md` under `plugins/` and `docs/`, every string in every +`plugin.json` and `marketplace.json` outside the `version` field, `plugins/*/hooks/*.sh`, +`.github/**`, `scripts/`, `.claude/`) found only third-party versions no local manifest can +adjudicate, plus claim shapes a manifest compare reads *wrong*: historical +("before the `0.6.0` split"), minimum floors ("implementation `0.9.0`+", true and documented +as explicitly *not* a version dependency), and planned values ("CREATE: `0.1.0`"). + +The **path guard** was built, tested to 61 contract cases, reviewed, and then withdrawn on +measurement. Swept across all 975 tracked markdown files, each fed to the hook as a real +`PostToolUse` payload: + +| | | +|---|---| +| Files firing | **231 / 975 = 23.7%** | +| Findings | **389** | +| True positives | **0** | + +The oracle never misfired — no candidate resolved at the repo root. Every finding was a +scoping failure. 72% were consumer-project config paths (`.claude/**` and similar) that a +doc describes for a *consuming* repo and that a marketplace correctly lacks; its +first-segment gate passed only because this repo happens to carry same-named top-level +directories. Fixing the three dominant causes still left ~4% firing at zero true positives. + +The **reference guard** shipped. Same corpus: 4 true positives, verified individually. Its +noise was 89% CHANGELOG rename entries — content the hook's own advisory calls correct as +written, because a CHANGELOG is append-only by contract and a rename entry must keep naming +the old command. Excluding CHANGELOGs moved it from 3.4% firing at 6% precision to **0.51% +firing at 57% precision**. + +## Decision + +**A verification guard does not ship default-on until its firing rate and precision have +been measured against a real corpus.** A sound oracle is a necessary condition, never a +sufficient one. Specifically: + +1. **Measure before shipping, on the real corpus, at real scale.** Not a sample, not the + contract suite. A passing contract suite proves the oracle; only a corpus sweep proves + the scoping. The path guard had 61 green cases and a 23.7% real-world firing rate. +2. **Report the number in the PR.** "It looks quieter now" is not a measurement. The + before/after firing rate and precision are the artifact that justifies default-on. +3. **Zero true positives on a real corpus disqualifies the guard**, however sound its + oracle. A guard that fires on a quarter of writes and is never right trains readers to + ignore every advisory, including the ones that are right — it has negative value, not + low value. +4. **Distinguish "wrong oracle" from "wrong scope."** The path guard's oracle was exact; its + scope was a repo whose docs describe other repos' trees. That distinction decides whether + a guard is deleted or re-filed for rescoping. It was re-filed (#1314). +5. **A guard whose only surface an existing gate owns does not ship at all.** Duplicating + `check-changelog-parity --check-bump` would have added a component that never fires + correctly — the defect class `PLUGIN-PHILOSOPHY.md` names as "a silently skipped feature + is a defect." + +## Consequences + +- Withdrawal is a normal outcome of the build, not a failure of it. Two of three candidates + in #1270 were withdrawn, one before implementation and one after full review. Both + withdrawals are recorded with their evidence (#1314 carries the sweep) so the ideas can be + rescoped rather than rediscovered. +- The cost is real: the path guard reached 61 contract cases and a full review round before + the measurement ran. Measuring earlier — right after the first working version, before + review — would have saved that. **The sweep belongs immediately after the guard first + works, not after it is polished.** +- This extends the verification-promotion discipline in + [ADR 0002](0002-default-on-ai-review-advisory-with-earned-promotion.md) one step earlier in + the lifecycle. That ADR governs promoting an advisory gate to blocking on demonstrated + precision; this one governs whether an advisory gate ships default-on at all, on the same + evidentiary basis. +- `docs/conventions/hook-precision/README.md` owns the over-fire discipline for a guard + already in the tree. This ADR is the pre-ship counterpart and defers to it thereafter. + +## Sources + +- #1270 (scoping, amended twice), #1284 (the two-guard PR, closed), #1319 (the shipped + guard), #1314 (the withdrawn guard's measurement and rescope) +- `docs/conventions/hook-precision/README.md` — over-fire discipline +- `docs/PLUGIN-PHILOSOPHY.md` — "a silently skipped feature is a defect" +- `melodic-software/standards`, `conventions/engineering/enforceability-tiers.md` — + classify the tier first, justify automation second From a94af104030113d9ab9c93ae5867108408bb5939 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 25 Jul 2026 12:45:10 -0400 Subject: [PATCH 2/5] docs(adr): ground the duplicate-gate rule in one-mechanism-per-concern MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review finding on #1357, and the misapplication was real. Rule 5 cited "a silently skipped feature is a defect" for rejecting a guard that duplicates an existing CI gate. That doctrine (`PLUGIN-PHILOSOPHY.md`'s prerequisites section) concerns a promised feature doing nothing when a runtime prerequisite is absent — `jq` missing from `PATH`. A duplicate gate is not that: it fires correctly, it simply adds no signal the required gate does not already produce. Grounded instead in **one mechanism per concern**, from the same document's validation section and owned upstream in `standards`. The correction matters because the criterion it hands a future guard author changes: "supplies no additional signal" is a coverage test, not a correctness one. As written it would have taught the wrong deletion rule. The same misapplication is corrected at its origin in #1270's body, where this ADR inherited it. Co-authored-by: Claude Opus 5 (1M context) --- ...-guards-earn-default-on-by-measured-precision.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md b/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md index 989256f4c1..6a056af33b 100644 --- a/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md +++ b/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md @@ -66,10 +66,13 @@ sufficient one. Specifically: 4. **Distinguish "wrong oracle" from "wrong scope."** The path guard's oracle was exact; its scope was a repo whose docs describe other repos' trees. That distinction decides whether a guard is deleted or re-filed for rescoping. It was re-filed (#1314). -5. **A guard whose only surface an existing gate owns does not ship at all.** Duplicating - `check-changelog-parity --check-bump` would have added a component that never fires - correctly — the defect class `PLUGIN-PHILOSOPHY.md` names as "a silently skipped feature - is a defect." +5. **A guard whose only surface an existing gate owns does not ship at all.** A guard + duplicating `check-changelog-parity --check-bump` would fire correctly; it would simply + add no signal the required gate does not already produce, while adding a second place the + rule can drift. That is the **one mechanism per concern** principle + (`PLUGIN-PHILOSOPHY.md`'s validation section, owned upstream in `standards`) — redundancy, + not a silent skip. The deletion criterion for a future guard author is "supplies no + additional signal," which is a coverage test, not a correctness one. ## Consequences @@ -94,6 +97,6 @@ sufficient one. Specifically: - #1270 (scoping, amended twice), #1284 (the two-guard PR, closed), #1319 (the shipped guard), #1314 (the withdrawn guard's measurement and rescope) - `docs/conventions/hook-precision/README.md` — over-fire discipline -- `docs/PLUGIN-PHILOSOPHY.md` — "a silently skipped feature is a defect" +- `docs/PLUGIN-PHILOSOPHY.md` — one mechanism per concern (validation section) - `melodic-software/standards`, `conventions/engineering/enforceability-tiers.md` — classify the tier first, justify automation second From 4d4d1edcd196794afc956c730dee5e1cbcf67fe7 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 25 Jul 2026 12:54:20 -0400 Subject: [PATCH 3/5] docs(adr): disqualify guards on observed false positives, not absent positives MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review finding on #1357, and a real logic flaw. Rule 3 said zero true positives on a real corpus disqualifies a guard. That categorically bars a correct guard for a rare defect: if the corpus contains no violation of the class, the guard stays silent, observes zero true positives, and precision is UNDEFINED rather than zero. The path guard's withdrawal was justified by 389 false findings, not by the absence of real ones. The rule now names the disqualifying condition as observed false positives with no true positive, and separates the three sweep outcomes explicitly: fires often with no true positives (disqualified), silent or near-silent (inconclusive on precision), fires rarely with true positives present (shippable). For a silent sweep the burden shifts to seeded defects — plant instances the guard should catch, confirm it catches them, and report the unseeded firing rate as the noise figure. Absence of evidence is not the evidence of absence rule 1 asks for. #1314's acceptance criteria carried the identical flaw and are corrected the same way, so the rescope it tracks is not held to an impossible bar. Co-authored-by: Claude Opus 5 (1M context) --- ...s-earn-default-on-by-measured-precision.md | 24 +++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md b/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md index 6a056af33b..497f06b3d0 100644 --- a/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md +++ b/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md @@ -59,10 +59,26 @@ sufficient one. Specifically: the scoping. The path guard had 61 green cases and a 23.7% real-world firing rate. 2. **Report the number in the PR.** "It looks quieter now" is not a measurement. The before/after firing rate and precision are the artifact that justifies default-on. -3. **Zero true positives on a real corpus disqualifies the guard**, however sound its - oracle. A guard that fires on a quarter of writes and is never right trains readers to - ignore every advisory, including the ones that are right — it has negative value, not - low value. +3. **A guard that fires and is never right disqualifies itself**, however sound its oracle. + The disqualifying condition is **observed false positives with no true positive**, not the + absence of true positives on its own. A guard that fires on a quarter of writes and is + never right trains readers to ignore every advisory, including the ones that are right — + negative value, not low value. + + **A silent guard is not thereby disqualified.** A corpus may simply contain no violation + of the class, which leaves precision *undefined* rather than zero, and a guard for a rare + defect is exactly the kind worth having. Distinguish the two outcomes: + + | Sweep result | Reading | + |---|---| + | fires often, no true positives | disqualified — the path guard, 389 findings, 0 real | + | silent, or near-silent | inconclusive on precision; establish detection by seeding known defects, then ship if the firing rate stays low | + | fires rarely, true positives present | shippable — the reference guard, 0.51% firing, 57% precision | + + For a silent sweep the burden shifts to **seeded defects**: plant instances the guard + should catch, confirm it catches them, and report the firing rate on the unseeded corpus + as the noise figure. Absence of evidence is not the evidence of absence that rule 1 asks + for. 4. **Distinguish "wrong oracle" from "wrong scope."** The path guard's oracle was exact; its scope was a repo whose docs describe other repos' trees. That distinction decides whether a guard is deleted or re-filed for rescoping. It was re-filed (#1314). From e20d3c578817343610235110067ebaf595d25d17 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 25 Jul 2026 13:44:54 -0400 Subject: [PATCH 4/5] docs(adr): require acceptable precision, and stop claiming a suite proves the oracle MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two review findings on #1357. Both are substantive, not wording. **Rarity plus one true positive was sufficient to ship.** The outcome matrix read "fires rarely, true positives present -> shippable", which one real finding among a hundred false ones satisfies — a 1%-precision guard, failing rule 3's substance for the same reason the path guard did. The row now requires rare firing AND acceptable precision. No numeric threshold is fixed: the tolerable ratio depends on how costly a false advisory is in the guarded surface, and a single number would be false precision. What is required is stating the measured precision, naming the ratio considered acceptable for that surface, and justifying it. The reference guard shipped at 57% with its residual noise attributed to one identified cause — that shape of argument, not the bare number, is what earns default-on. **A contract suite cannot prove an oracle.** It is evidence for the cases it contains; an input shape absent from the suite can still be misclassified. As written, rule 1 would have steered an author to diagnose every corpus finding as a scoping failure under rule 4, when it might be an oracle defect the suite had no case for. Rule 4 now says the distinction is established rather than assumed, and records how it was established for the path guard: each of the 389 candidates was re-tested against the repo root and confirmed genuinely absent. That check is what made "exact oracle, wrong scope" a finding instead of an inference. Co-authored-by: Claude Opus 5 (1M context) --- ...s-earn-default-on-by-measured-precision.md | 28 +++++++++++++++---- 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md b/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md index 497f06b3d0..de42d2519c 100644 --- a/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md +++ b/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md @@ -55,8 +55,10 @@ been measured against a real corpus.** A sound oracle is a necessary condition, sufficient one. Specifically: 1. **Measure before shipping, on the real corpus, at real scale.** Not a sample, not the - contract suite. A passing contract suite proves the oracle; only a corpus sweep proves - the scoping. The path guard had 61 green cases and a 23.7% real-world firing rate. + contract suite. A passing contract suite is evidence for the cases it contains — nothing + more; an input shape absent from the suite can still be misclassified. A corpus sweep is + what exposes the scoping. The path guard had 61 green cases and a 23.7% real-world firing + rate. 2. **Report the number in the PR.** "It looks quieter now" is not a measurement. The before/after firing rate and precision are the artifact that justifies default-on. 3. **A guard that fires and is never right disqualifies itself**, however sound its oracle. @@ -73,15 +75,29 @@ sufficient one. Specifically: |---|---| | fires often, no true positives | disqualified — the path guard, 389 findings, 0 real | | silent, or near-silent | inconclusive on precision; establish detection by seeding known defects, then ship if the firing rate stays low | - | fires rarely, true positives present | shippable — the reference guard, 0.51% firing, 57% precision | + | fires rarely AND precision is acceptable | shippable — the reference guard, 0.51% firing, 57% precision | For a silent sweep the burden shifts to **seeded defects**: plant instances the guard should catch, confirm it catches them, and report the firing rate on the unseeded corpus as the noise figure. Absence of evidence is not the evidence of absence that rule 1 asks for. -4. **Distinguish "wrong oracle" from "wrong scope."** The path guard's oracle was exact; its - scope was a repo whose docs describe other repos' trees. That distinction decides whether - a guard is deleted or re-filed for rescoping. It was re-filed (#1314). + + **"Acceptable precision" is a judgment, and the ADR deliberately does not fix a + threshold** — the tolerable ratio depends on how costly a false advisory is in the surface + being guarded, and a single number would be false precision. But rarity plus one true + positive is explicitly NOT sufficient: one real finding among a hundred false ones is a + 1%-precision guard however rarely it fires, and it fails rule 3's substance for the same + reason the path guard did. State the measured precision, name the ratio you consider + acceptable for that surface, and justify it. The reference guard shipped at 57% with the + remaining noise attributed to a single identified cause — that shape of argument, not the + bare number, is what earns default-on. +4. **Distinguish "wrong oracle" from "wrong scope" — by checking, not by assuming.** That + distinction decides whether a guard is deleted or re-filed for rescoping, so it is worth + establishing rather than inferring from a green suite. For the path guard it was + established per finding: each candidate was re-tested against the repo root and confirmed + genuinely absent, which is what made "the oracle is exact, the scope is wrong" a finding + instead of an assumption. It was re-filed (#1314). Had any candidate turned out to resolve, + that would have been an oracle defect the contract suite simply had no case for. 5. **A guard whose only surface an existing gate owns does not ship at all.** A guard duplicating `check-changelog-parity --check-bump` would fire correctly; it would simply add no signal the required gate does not already produce, while adding a second place the From a699f496f7c35aa0bc8f66854b77f3990f5cb982 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Sat, 25 Jul 2026 13:53:41 -0400 Subject: [PATCH 5/5] docs(adr): scope the no-firing exemption to zero findings, not low volume MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review finding on #1357, and my own wording contradicted the rule two lines above it. The matrix row read "silent, or near-silent -> inconclusive on precision", which permitted shipping a guard that produced one false finding and no real ones — 0% precision, exactly what rule 3 disqualifies. "Near-silent" smuggled a low-volume allowance into what was meant to be a zero-findings exemption. The exemption is now explicitly for **zero findings only**, and the rows state why: precision is undefined when a guard never fires, but it is 0% whether the firing count is 389 or 1. Scarcity changes the volume of noise, never its ratio. Also corrects two phrases the old row had infected: the seeded-defect paragraph now says "zero-finding sweep" rather than "silent sweep", and the acceptable-precision paragraph now argues from the ratio being scale-invariant rather than implying rarity is a partial defence. Co-authored-by: Claude Opus 5 (1M context) --- ...s-earn-default-on-by-measured-precision.md | 30 ++++++++++--------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md b/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md index de42d2519c..d1ccc889fd 100644 --- a/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md +++ b/docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md @@ -67,27 +67,29 @@ sufficient one. Specifically: never right trains readers to ignore every advisory, including the ones that are right — negative value, not low value. - **A silent guard is not thereby disqualified.** A corpus may simply contain no violation - of the class, which leaves precision *undefined* rather than zero, and a guard for a rare - defect is exactly the kind worth having. Distinguish the two outcomes: + **A guard that does not fire at all is not thereby disqualified.** A corpus may simply + contain no violation of the class, which leaves precision *undefined* rather than zero, and + a guard for a rare defect is exactly the kind worth having. The exemption is for **zero + findings only** — it is not a low-volume allowance, because precision does not improve with + scarcity: | Sweep result | Reading | |---|---| - | fires often, no true positives | disqualified — the path guard, 389 findings, 0 real | - | silent, or near-silent | inconclusive on precision; establish detection by seeding known defects, then ship if the firing rate stays low | - | fires rarely AND precision is acceptable | shippable — the reference guard, 0.51% firing, 57% precision | + | **zero findings** — did not fire at all | precision *undefined*, not zero. Inconclusive: establish detection by seeding known defects, then ship if the unseeded firing rate stays acceptable | + | fires at all, no true positive | disqualified. Precision is 0% whether the firing count is 389 or 1 — the path guard is the loud case, but a single false finding with no real one fails identically | + | fires AND precision is acceptable | shippable — the reference guard, 0.51% firing, 57% precision | - For a silent sweep the burden shifts to **seeded defects**: plant instances the guard - should catch, confirm it catches them, and report the firing rate on the unseeded corpus - as the noise figure. Absence of evidence is not the evidence of absence that rule 1 asks - for. + For a zero-finding sweep the burden shifts to **seeded defects**: plant instances the guard + should catch, confirm it catches them, and report the unseeded firing rate — which is zero + by construction in this branch — as the noise figure. Absence of evidence is not the + evidence of absence that rule 1 asks for. **"Acceptable precision" is a judgment, and the ADR deliberately does not fix a threshold** — the tolerable ratio depends on how costly a false advisory is in the surface - being guarded, and a single number would be false precision. But rarity plus one true - positive is explicitly NOT sufficient: one real finding among a hundred false ones is a - 1%-precision guard however rarely it fires, and it fails rule 3's substance for the same - reason the path guard did. State the measured precision, name the ratio you consider + being guarded, and a single number would be false precision. But rarity is not itself + evidence: one real finding among a hundred false ones is a 1%-precision guard however + rarely it fires, and it fails for the same reason the path guard did at 23.7%. Scarcity + changes the volume of noise, never its ratio. State the measured precision, name the ratio you consider acceptable for that surface, and justify it. The reference guard shipped at 57% with the remaining noise attributed to a single identified cause — that shape of argument, not the bare number, is what earns default-on.