docs(specs): record the D1 no-op-detector measurement and verdict - #3188
Conversation
#3121 asked whether cut class D1 — "content the model already knows" — is a scanner shape, a judgment shape, or a routing finding, and required the answer to cite a false-positive rate measured on a real sample rather than estimated. Measured it: 895 agent-facing markdown files, 13,529 instruction sentences, the proposed predicate implemented verbatim. It flags 6,107 of them (45.1%). A seeded, stratified sample of 185 flagged sentences, hand-adjudicated against the protected-content list, yields a 94.1% false-positive rate scoring every contested call in the proxy's favour, and 100% scoring them against it. Zero unambiguous true positives. The failure is directional, not just noisy: 57.0% of the flagged population is in hard-boundary register, because the house style writes its load-bearing rules as bare imperatives. Over this corpus the proxy is closer to an inverse detector for its target class. Verdict: routing finding — D1 belongs to claude-config:unhobble, which measures the model, not to a text scanner or a model-graded lane that grades text. The boundary is one audit-instructions already states in its own Scope section. Records the method precisely enough to reimplement, the results, and all 185 adjudicated rows, so the 94.1% bar #3124 is held to stays checkable. Closes #3121
|
Claude finished @kyle-sexton's task in 3m 39s —— View job Claude finished the code review
SummaryThis PR adds a single docs-only file, Verification performedPer the code-review lane's rule against restating author-claimed verification, I independently checked what I could from the diff itself rather than trusting the PR description:
FindingsNone. No correctness or maintainability issues that would block this PR — the arithmetic is internally consistent, the appendix reconciles with the summary tables, and every sampled quote traces back to real source text. The doc-only nature of the change and its explicit "no production code" scope (confirmed against |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: db05375646
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Claude has reviewed this PR 4 times. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
Codex review on #3188 found the record could not actually be reproduced from its own prose, and it was right on all three counts. The revision was unpinned. The corpus selectors are relative to a working tree, so they drift as the fleet grows: applying them to current main yields 904 files against the 895 measured. The measurement was taken at dff0942, the session's checkout — now stated, with the reason the pin is load-bearing. The fixed lists were summarised rather than given. IMPERATIVE_OPENERS (125 verbs), NOT_PROPER (197 words), ABBREV, and PATHISH's extension set each change what gets flagged, so "a fixed opener list" does not determine the 13,529 / 6,107 totals the way the prose implied. The sampling order was omitted. The draw is order-sensitive; the seed alone does not fix the rows without the (file, sentence) sort. Rather than enumerate all of it in prose, the harness is committed under docs/specs/d1-model-already-knows-measurement/ with a reproduction recipe. Verified end-to-end against the pinned revision in a clean worktree: 895 files, 13,529 instruction sentences, 6,107 flagged (45.1%), the same 185 sampled rows in the same order, 94.1% / 100%, 57.0% hard-boundary register. This matters because #3124's acceptance criteria bind any future implementation to the 94.1% bar, and a bar nobody can recompute is not a bar. Refs #3121
|
Claude finished @kyle-sexton's task in 4m 34s —— View job Code review complete
SummaryThis commit responds to Codex's prior reproducibility finding by pinning the measured revision and committing the harness ( Findings (posted inline)Two internal-consistency defects, both concrete and checkable from the diff itself — the doc's prose defines a test, the shipped code implements a materially different one, and the doc's key numbers derive from the code, not the prose:
Everything else checked out: arithmetic in the flag-rate and adjudicated-sample tables is internally consistent, No other correctness or maintainability issues found in the new scripts. Branch · |
The hygiene job's exec-bit check went red on 5ce7eb0: all three harness scripts carry a `#!/usr/bin/env python3` shebang but were committed 100644. Every other shebang-bearing .py tracked in this repo is 100755, and these three were the only shebang+644 files in the tree — so this is the repo's existing convention, not a new one. Mode change only; the blob hashes are unchanged, so the reproduction verified at dff0942 still holds. Every other hygiene check on that run was green (markdown, typos, gitleaks, editorconfig, shellcheck, comment-hygiene, machine-specific-paths, eol-renormalize, schemas, hook-wiring-liveness). Refs #3121
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Claude finished @kyle-sexton's task in 4m 16s —— View job Claude finished the code review
SummaryThis push responds to Codex's P2 from the first commit (unpinned revision, summarized-not-shipped word lists, unfixed sampling order) by committing the harness ( Verification performedThis lane's Bash access is scoped to
FindingsNone. Everything I could check without executing the harness — arithmetic, appendix-to-verdict-set reconciliation, script/README argument agreement, exec-bit fix — is internally consistent, and the specific gap I went looking for in the segmentation code doesn't have a live instance in the corpus it scans. I was not able to independently re-run the harness end-to-end (this lane's Bash access doesn't extend to arbitrary script execution), so I can't personally confirm the headline 895/13,529/6,107/94.1%/100%/57.0% figures — that verification is author-claimed, per kyle-sexton's reply describing a clean detached-worktree rerun at the pinned revision. |
…bset Automated review found the Unit section's modal-directive list claimed 13 tokens where MODALS matches 20 — omitting shall, require, requires, may not, can't, needs to, and be sure. Not cosmetic: that test gates whether a sentence enters the 13,529-sentence denominator at all, so "Configuration changes shall be reviewed." is counted by the script and would not be by a reader reimplementing the documented rule. Same defect class as the reproducibility finding two commits ago — prose drifting from the shipped predicate in a record whose whole purpose is letting #3124 recompute the 94.1% bar. Fixed the same way: name the constant rather than restate it, matching how IMPERATIVE_OPENERS, NOT_PROPER, ABBREV, and PATHISH are already handled. The one place an inline enumeration survived is the one place that drifted. Swept the rest of the Method section against the code while here: the six signal families, the 12 PATHISH extensions, the 125 openers, and the 12-char fragment floor all match. Refs #3121
Automated review found the register regex broader than what the record said it
measured. It attributed the 57.0% figure to four words but also matched bare
"no " — which fires on ordinary prose ("no proper noun", "no path", "there is
no reason to") and is not a boundary marker.
Measured the difference rather than assuming it small:
57.0% as shipped, including bare "no "
54.8% never | must | do not | don't | cannot <- corrected
50.5% strictly the four words the doc named
2.2pt contributed by bare "no " alone, across 134 sentences
Dropped bare "no " and kept don't, which is a genuine boundary marker the
prose had omitted; the doc and README now state the exact token set and point
at the constant. The figure is 54.8%.
The argument is unchanged: a clear majority of the flagged population is still
hard-boundary register, which is the claim the number supports. The 94.1%
false-positive rate — the bar #3124 is held to — is untouched, since it comes
from the hand adjudication and not from this regex.
Third finding in this PR of one class: prose asserting something the shipped
code did not do. The first two were unverifiable claims; this one moved a
published number, which is why it is worth stating plainly rather than
quietly amending.
Refs #3121
|
Claude finished @kyle-sexton's task in 1m 30s —— View job Code review complete
SummaryThis update (commits Verification performed
FindingsNone. Both P2/informational issues from the prior review round are substantively fixed, not just reworded — the doc's stated method now matches the shipped code for both the modal-directive test and the hard-boundary regex, and the updated 54.8% figure is propagated consistently everywhere it's cited. |
Closes #3118 ## Summary Closes the #3118 work-map by recording what each child actually shipped. The brief lived in the issue body; the spec is the durable map so a later reader does not have to reconstruct it from closed children. ## Fix Adds `docs/specs/agent-doc-register-detectors.md` and points the D1 measurement spec at it: - D1 (#3124): closed unbuilt; measurement lives in `unhobble` (#3188) - D2 (#3120): `audit-instructions` I28 coercive-emphasis / blanket-tool-default - D3 (#3123): `docs-hygiene:audit-noise` negation-without-positive - D4 (#3186): sibling restatement — separate PR Docs-only; no plugin version bump, so it cannot collide with #3186's `claude-config` detector work. ## Verification - Spec exists and matches shipped rule ids - D1 measurement spec now points at this register - `check-lane-coverage.sh --check` — clean on this branch - markdownlint on the two changed specs — 0 issues ## Related - Refs #3120, #3121, #3123, #3124, #3186, #3188 Co-authored-by: Kyle Sexton <kyle-sexton@users.noreply.github.com>
2 participants
Closes #3121
Summary
#3121 asked whether cut class D1 — content the model already knows — is a scanner shape, a judgment shape, or a routing finding, and required the answer to cite a false-positive rate measured on a real sample rather than estimated. This lands that measurement, its harness, and the verdict as a durable spec record.
The verdict is routing finding: D1 belongs to
claude-config:unhobble, which measures the model, not to a text scanner and not to a model-graded lane that grades text.Fix
Adds
docs/specs/d1-model-already-knows-measurement.md, a sibling of the existing briefs in that directory, recording:Adds
docs/specs/d1-model-already-knows-measurement/— the three-stage harness (d1_proxy.py,sample.py,adjudication.py) plus a README with the reproduction recipe.No production code, per the scope #3121 set for itself. Nothing under
plugins/is touched; the added Python is measurement apparatus for a spec record, not a plugin, skill, or CI script.Verification
The measurement, taken against
dff0942917e56929f6146261117a0eceeac502c8:random.Random(3121), stratifiedThe failure is directional rather than merely noisy: 54.8% of the flagged population is in hard-boundary register (
never,must,do not,don't,cannot), because the house style writes its load-bearing rules as bare imperatives. Sentences the predicate flags includeA human merges — this skill never auto-mergesandDo not silently fall back to training data. Over this corpus the proxy is closer to an inverse detector for its target class, and #3124's remediation is whole-sentence deletion.Reproduction proven, not asserted. The harness was re-run end-to-end in a clean detached worktree at the pinned revision: 895 files, 13,529 sentences, 6,107 flagged, the same 185 sampled rows in the same order, 94.1% / 100%.
Gates run locally against the source checkout:
ruff check/ruff format --check— all checks passed, 3 files already formattedmarkdownlint-cli2 --config .markdownlint-cli2.jsonc— 0 issuestypos --config _typos.toml— cleaneditorconfig-checker— cleangitleaks detect --config .gitleaks.toml— no leaksscripts/check-detector-findings-crosswalk.sh --check— exit 0scripts/check-fleet-audit-doc-grammar.sh --check— exit 0Review history
Three findings, all correct, all fixed. They are one class: prose asserting something the shipped code did not do, in a record whose whole purpose is making a number checkable.
dff09429(the session checkout), neverorigin/main; the fixed word lists were summarised rather than shipped; the sampling order was omitted, soseed=3121alone did not fix the rows. Fixed in5ce7eb08by committing the harness and pinning the revision.MODALSmismatch. The doc listed 13 modal tokens where the code matches 20, omittingshall,require,requires,may not,can't,needs to,be sure— the test gating entry to the 13,529-sentence denominator. Fixed in41863bbbby naming the constant rather than restating it.no, which fires on ordinary prose (no proper noun,no path). Measured: 57.0% as shipped, 54.8% with the corrected token set, 2.2 points contributed by barenoalone across 134 sentences. Fixed ind7eb8cc0; the figure is 54.8%.The verdict is unchanged by all three. A clear majority of the flagged population is still hard-boundary register, which is what makes the failure directional rather than merely noisy. 94.1% is untouched — it comes from the hand adjudication of the 185 sampled sentences, not from any regex — so the bar #3124 is held to does not move.
One latent issue was raised and deliberately not filed:
segment()would misread a 4-space-indented ordered-list item as code and drop it. Checked all 895 corpus files at the pinned revision — zero instances, so it moved no reported number.Related