docs(adr): record that verification guards earn default-on by measured precision - #1357
Conversation
…d precision Closes #1352 The #1270 program scoped three claim-verification guards on sound oracles and shipped one. The generalizable finding — a sound oracle is necessary but not sufficient — lived only in closed PRs and issue bodies. The asserted-path guard reached 61 green contract cases and a full review round before a corpus sweep showed 23.7% of files firing across 389 findings with zero true positives. Its oracle was exact; its scope was a repo whose docs mostly describe other repos' trees. A passing contract suite proves the oracle; only a corpus sweep proves the scoping. Records four rules: measure against the real corpus before shipping default-on, report the number in the PR, treat zero true positives as disqualifying however sound the oracle, and distinguish wrong-oracle from wrong-scope because that decides deletion versus rescoping. Also records the cost — measuring right after the guard first worked, instead of after polish and review, would have saved that review round. Extends ADR 0002's verification-promotion discipline one step earlier in the lifecycle: 0002 governs promoting an advisory gate to blocking on demonstrated precision, this governs whether it ships default-on at all, on the same evidentiary basis. Defers to docs/conventions/hook-precision/README.md for the post-ship over-fire discipline rather than restating it. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 2m 8s —— View job PR Review
This is a docs-only PR adding ADR 0003. All in-repo cross-references were verified by file lookup; the quote from Format and structureMatches the house format ( Cross-references
Data consistencyThe path guard table at FindingsNit — inconsistent script reference Context (line 22) writes: Decision rule 5 (line 70) writes: The actual script is Nit — empty table header row The path guard measurement table at line 33–38 uses an empty Content assessmentThe five decision rules (measure first, report the number, zero true positives disqualifies, distinguish oracle vs. scope, surface already covered by an existing gate does not ship) are well-grounded in the evidence presented. The cost accounting in Consequences — measuring after polish rather than immediately after first-working cost a full review round — is the most actionable takeaway and is stated precisely. The relationship to ADR 0002 is correctly scoped: that ADR governs advisory→blocking promotion; this one governs whether a guard ships default-on at all. No duplication. Approved. Two nits above are both optional cleanup — neither is a correctness issue. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f7e2fb23e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Review finding on #1357, and the misapplication was real. Rule 5 cited "a silently skipped feature is a defect" for rejecting a guard that duplicates an existing CI gate. That doctrine (`PLUGIN-PHILOSOPHY.md`'s prerequisites section) concerns a promised feature doing nothing when a runtime prerequisite is absent — `jq` missing from `PATH`. A duplicate gate is not that: it fires correctly, it simply adds no signal the required gate does not already produce. Grounded instead in **one mechanism per concern**, from the same document's validation section and owned upstream in `standards`. The correction matters because the criterion it hands a future guard author changes: "supplies no additional signal" is a coverage test, not a correctness one. As written it would have taught the wrong deletion rule. The same misapplication is corrected at its origin in #1270's body, where this ADR inherited it. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 1m 46s —— View job PR Review (updated — commit a94af10)
The second commit ( Codex P2 — misapplied doctrine in rule 5Addressed. The old rule 5 said a duplicate guard would "never fire correctly" and cited the silently skipped feature doctrine from The rewrite at lines 69–75 corrects both:
The Sources entry is updated to match: The rewrite is more accurate and more useful to future guard authors — it names the right diagnostic question ("does this supply additional signal?") rather than the wrong one ("does this fire correctly?"). Nits from prior review — status unchangedBoth remain. Neither is a correctness issue and neither is a blocker. Nit — inconsistent script reference Line 21 uses Nit — empty table header row Lines 33–34 use SummaryThe substantive Codex finding is fully resolved. No blocking issues. The ADR's five rules are well-grounded in their evidence, the cross-references all verify, and rule 5 now teaches the correct mental model. Approved. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a94af10403
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…positives Review finding on #1357, and a real logic flaw. Rule 3 said zero true positives on a real corpus disqualifies a guard. That categorically bars a correct guard for a rare defect: if the corpus contains no violation of the class, the guard stays silent, observes zero true positives, and precision is UNDEFINED rather than zero. The path guard's withdrawal was justified by 389 false findings, not by the absence of real ones. The rule now names the disqualifying condition as observed false positives with no true positive, and separates the three sweep outcomes explicitly: fires often with no true positives (disqualified), silent or near-silent (inconclusive on precision), fires rarely with true positives present (shippable). For a silent sweep the burden shifts to seeded defects — plant instances the guard should catch, confirm it catches them, and report the unseeded firing rate as the noise figure. Absence of evidence is not the evidence of absence rule 1 asks for. #1314's acceptance criteria carried the identical flaw and are corrected the same way, so the rescope it tracks is not held to an impossible bar. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 1m 30s —— View job PR Review (updated — commit 4d4d1ed)
The third commit ( Codex P2 — reject guards based on false positives, not absent positivesAddressed. The Codex finding was precise: the old rule 3 conflated "zero true positives" with "disqualified," which would have barred a correct guard that runs silently against a corpus that happens to contain no violations. The rewrite at lines 62–81 corrects this precisely:
The Sources entry is unchanged and correct. ✓ The rewrite gives future guard authors the correct three-way diagnostic: fires-and-wrong (delete or rescope), silent (seed and measure), fires-and-right (ship) — which is more useful than the binary it replaced. Nits from prior reviews — status unchangedBoth remain. Neither is a correctness issue and neither is a blocker. Nit — inconsistent script reference Line 21 uses Nit — empty table header row Lines 33–34 use SummaryAll three Codex P2 findings across the PR's history are now resolved. No blocking issues remain. The ADR's five decision rules are grounded in their evidence, all cross-references verify, and rule 3 now teaches the correct three-way diagnostic rather than the binary it replaced. Approved. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4d4d1edcd1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…oves the oracle Two review findings on #1357. Both are substantive, not wording. **Rarity plus one true positive was sufficient to ship.** The outcome matrix read "fires rarely, true positives present -> shippable", which one real finding among a hundred false ones satisfies — a 1%-precision guard, failing rule 3's substance for the same reason the path guard did. The row now requires rare firing AND acceptable precision. No numeric threshold is fixed: the tolerable ratio depends on how costly a false advisory is in the guarded surface, and a single number would be false precision. What is required is stating the measured precision, naming the ratio considered acceptable for that surface, and justifying it. The reference guard shipped at 57% with its residual noise attributed to one identified cause — that shape of argument, not the bare number, is what earns default-on. **A contract suite cannot prove an oracle.** It is evidence for the cases it contains; an input shape absent from the suite can still be misclassified. As written, rule 1 would have steered an author to diagnose every corpus finding as a scoping failure under rule 4, when it might be an oracle defect the suite had no case for. Rule 4 now says the distinction is established rather than assumed, and records how it was established for the path guard: each of the 389 candidates was re-tested against the repo root and confirmed genuinely absent. That check is what made "exact oracle, wrong scope" a finding instead of an inference. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 1m 29s —— View job PR Review (updated — commit e20d3c5)
The fourth commit ( Codex P2 — require acceptable precision before calling a guard shippableAddressed. The prior matrix row read "fires rarely, true positives present" — Codex correctly identified that one TP among 99 FPs would pass that test. Two changes fix this:
Codex P2 — avoid claiming a contract suite proves the oracleAddressed. Lines 58–60 now read:
The word "proves" is gone and the epistemic limit is stated precisely. The correction is consequential for rule 4: if an author mistakenly believed a green suite proved the oracle, they would diagnose all corpus findings as scoping failures without checking — which is exactly the error rule 4 guards against ("by checking, not by assuming"). Nits from prior reviews — status unchangedBoth remain. Neither is a correctness issue and neither is a blocker. Nit — inconsistent script reference Line 21 uses Nit — empty table header row Lines 33–34 use SummaryAll five Codex P2 findings across the PR's four commits are now resolved. No blocking issues remain. The five decision rules are grounded in their evidence, all cross-references verify, and the precision paragraph gives future guard authors a richer mental model than a fixed threshold would: the shape of argument (precision measured, ratio justified for the surface, noise attributed) is what earns default-on. Approved. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e20d3c5788
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…lume Review finding on #1357, and my own wording contradicted the rule two lines above it. The matrix row read "silent, or near-silent -> inconclusive on precision", which permitted shipping a guard that produced one false finding and no real ones — 0% precision, exactly what rule 3 disqualifies. "Near-silent" smuggled a low-volume allowance into what was meant to be a zero-findings exemption. The exemption is now explicitly for **zero findings only**, and the rows state why: precision is undefined when a guard never fires, but it is 0% whether the firing count is 389 or 1. Scarcity changes the volume of noise, never its ratio. Also corrects two phrases the old row had infected: the seeded-defect paragraph now says "zero-finding sweep" rather than "silent sweep", and the acceptable-precision paragraph now argues from the ratio being scale-invariant rather than implying rarity is a partial defence. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 1m 24s —— View job PR Review (updated — commit a699f49)
The fifth commit ( Codex P2 — separate silent sweeps from low-volume false positivesAddressed. The prior matrix had a "near-silent" row that covered both zero-findings and low-volume false positives, allowing a guard with one false finding and no true positives to reach the seeded-defects path and potentially ship. The Codex finding was precise: precision is 0% the moment any false finding exists, not inconclusive — so that case must be disqualified, not redirected. Lines 70–85 correct this cleanly:
The commit title mirrors the substance exactly: "scope the no-firing exemption to zero findings, not low volume." Nits from prior reviews — status unchangedBoth remain. Neither is a correctness issue and neither is a blocker. Nit — inconsistent script reference Line 21 uses Nit — empty table header row Lines 33–34 use SummaryAll six Codex P2 findings across the PR's five commits are now resolved. The ADR's five decision rules are grounded in their evidence, all cross-references verify, and the precision/scoping logic is now watertight:
Approved. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a699f496f7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| The **path guard** was built, tested to 61 contract cases, reviewed, and then withdrawn on | ||
| measurement. Swept across all 975 tracked markdown files, each fed to the hook as a real | ||
| `PostToolUse` payload: |
There was a problem hiding this comment.
For ordinary PostToolUse:Edit events, the repository's precision convention requires scanning only the changed hunk (docs/conventions/hook-precision/README.md, lines 20–22), whereas feeding every complete tracked file through the hook measures static file prevalence under synthetic whole-file writes. Consequently, the reported 23.7% cannot be treated as a real-world firing rate: files containing an old candidate do not fire when unrelated hunks are edited, and actual edit frequency is absent. Replay representative Write/Edit payloads, or label this metric as whole-file prevalence rather than using it as the firing-rate evidence required by the ADR.
Useful? React with 👍 / 👎.
1 participant
Closes #1352
Summary
Adds
docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md, recording the reusable lesson from the #1270 guard program. Docs-only.Three guards were scoped on sound oracles. One shipped (#1319, 0.51% firing at 57% precision). Two were withdrawn:
scripts/check-changelog-parity.sh --check-bump, and the residual prose surface is historical, minimum-floor, and planned version claims a manifest compare reads wrong.The generalizable finding is that a sound oracle is necessary but not sufficient. The path guard's oracle was exact � no candidate resolved at the repo root, every finding was a scoping failure. A passing contract suite proves the oracle; only a corpus sweep proves the scoping.
Four rules recorded: measure against the real corpus before shipping default-on; report the number in the PR; treat zero true positives as disqualifying however sound the oracle; distinguish wrong-oracle from wrong-scope, because that decides deletion versus rescoping.
It also records the cost honestly � measuring immediately after the guard first worked, rather than after polish and a review round, would have saved that round. That is the actionable part for whoever builds the next one.
Test plan
markdownlint-cli2on the new file � 0 errorsdocs/adr/0002-�md,docs/conventions/hook-precision/README.md,docs/PLUGIN-PHILOSOPHY.md,scripts/check-changelog-parity.sh# Title, Status/Date bullets, Context, Decision, Consequences)Related
Extends ADR 0002 one step earlier in the lifecycle � 0002 governs promoting an advisory gate to blocking on demonstrated precision, this governs whether it ships default-on at all, on the same evidentiary basis. Defers to
docs/conventions/hook-precision/README.mdfor post-ship over-fire discipline rather than restating it.#1270 (scoping, amended twice), #1319 (shipped guard, merged), #1284 (closed), #1314 (withdrawn guard and its measurement).