Assert a floor on what a diff-scoped prose run actually scanned - #594
Merged
Conversation
A gate that finds nothing is indistinguishable from a gate with nothing to find, and every verdict a --diff run reports rests on a scope it never checked. Four routes to the same false clean are already on record, each closed by a guard written after a reviewer noticed it: an unresolvable base widening to a whole-tree scan, a multi-line paths input read only to its first newline, a diff taken in one repository while scanning another, and a path under no repository at all. The fifth route gets found the same way or not at all, which is why this asserts a floor rather than adding a guard. A run that resolves a non-empty diff and matches none of the files it names now exits 2 naming them, rather than exiting 0. Zero alone is not the test, since a change touching only files the rules do not read, an image or a lock file, legitimately matches nothing, so the comparison is against the diff's own list of files this run could have read. The floor found the fifth route immediately. git diff reports repository-relative keys while discovery keys off the directory the run started in, so running the gate from a subdirectory intersects to nothing and reported clean. unread_diff_files resolves against the repository top level for that reason: reading the diff's paths against the working directory would leave the list empty in exactly the case it exists to catch. Ten cases lock it. Three fail against the previous revision, and seven pass against both because they hold the other direction, that a deliberate narrowing, an empty diff, an excluded file, a generated tree, a deletion and a binary are each a clean run rather than a scoping failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR strengthens the reliability of scripts/prose_lint.py --diff by asserting a minimum “diff scope floor”: if a non-empty diff is resolved but the scan matches none of the diff’s readable, in-scope files, the run now fails (exit code 2) rather than reporting a false clean.
Changes:
- Add
asked_about()andunread_diff_files()helpers to determine which diff-named files the run was asked to cover and could have read. - Update
main()to exit 2 (with a targeted error message) when a non-empty diff yields zero matched files but the diff names readable files that should have been scanned. - Add a new
TestDiffScopeFloorsuite covering both “must fire” and “must not fire” scenarios (including the subdirectory-run key mismatch).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| scripts/prose_lint.py | Implements the diff-scope floor check and supporting helpers, returning exit code 2 on confirmed scoping failures. |
| scripts/test_prose_lint.py | Adds focused unit tests validating the new floor behavior and guarding legitimate zero-match cases. |
This was referenced Aug 7, 2026
ptr727
added a commit
that referenced
this pull request
Aug 7, 2026
## Why `GOVERNANCE.md` "Representative Data in Agent-Authored Text" says agent-authored text illustrates with constructed data rather than data observed in the maintainer's environment, and says in the same breath that no pattern closes it. This gates the half a pattern can reach. It is introduced as that half deliberately. The leak that motivated the rule was name-shaped and sat in a pull request comment, which no committed-file linter reads, so a check mistaken for the answer is what stops anyone looking at the exposure the rule actually exists for. Carries the `TODO.md` cluster "The Representative-Data Path Check", which #595 released from `blocked` once the scope floor shipped in #594. ## The shapes were measured, not assumed The backlog recorded four shapes, `/home/<name>`, `/Users/<name>`, `C:\Users\<name>`, and a bare drive letter. Run against this repository, **a bare drive letter matched 11 files and named a path in none of them.** An escaped newline after any word ending in a letter and a colon reads as a drive letter, so a YAML fixture is full of them: | Matched text | What it actually is | | --- | --- | | `e:\n needs` | `jobs:\n needs` inside a test fixture string | | `s:\n a` | `jobs:\n a:` inside a test fixture string | | `C:\tmp` (8 hits) | a temp path in fixtures for the string-literal rule | That shape is dropped. The one kept is a drive letter followed by `Users`, which matches nothing in the tree today. This is narrower than the committed prose, which says "an absolute home path or a drive letter", so the narrowing is stated here rather than left for a reader to discover. ## The exemption is built, not listed A wrong exemption hands out a work list that damages correct documents, and this rule has two self-referential problems that a file-name allowlist would solve badly and briefly. **The rule's own wording quotes the shapes it forbids.** Requiring a real user segment solves it by construction: the documentation placeholder is not a valid account name, so it describes the shape without matching it. No exemption names `GOVERNANCE.md`, and none goes stale when the text moves. | Probe | Verdict | | --- | --- | | `C:\Users\<name>` in documentation | no match | | `/home/<name>` in documentation | no match | | `/home/vscode/.ssh` | exempt, container account | | a real absolute home path | **flagged** | **Container accounts are exempt by name.** Every home path in this repository, all 15, is `/home/vscode` from the devcontainer snippets and the doc describing them. Without that exemption the rule would open with 15 findings and no true positive among them. **The gate reads its own source**, so the bait paths in the tests are assembled from parts and a case asserts that neither `prose_lint.py` nor its test file carries a literal. That is the approach the file already takes for non-ASCII, which it writes as escapes for exactly this reason. ## Operational repositories are exempt, and told so An operational repository's runbook carries the literal path an operator types. That is the repository's own content rather than an agent quoting an environment it observed, which is the distinction the rule is about. The model is read from what the repository carries rather than from the hub registry, which a downstream checkout cannot see: `spec/files.json` declares `repo-config/operational/develop.json` for the operational model and `repo-config/develop.json` for the release one. The hub carries both, being the template for each, so it reads as release rather than exempting itself from a rule it authors. **The skip is announced on stderr**, because a rule that silently stops running reads as a rule that passed, which is the same failure #594 closed on the other side. ## Scope note Unlike every other prose rule, this one reads fenced blocks and configuration values rather than stripped prose. A transcript pasted from a terminal and a bind mount naming a real home are the exposure in its most consequential forms, and neither is a sentence. ## Verification - `python3 scripts/test_prose_lint.py` - 188 pass, up from 173. Of the 14 new cases, 10 fail against `develop` and 4 pass against both, holding the must-not-fire direction. - `python3 scripts/prose_lint.py . --check home-path` - clean across the tracked tree. - `python3 scripts/prose_lint.py . --summary` - 557 violations across 45 files, the #519 backlog figure unchanged, with `home-path` contributing none. Adding a rule to `DEFAULT_RULES` did not move the number. - `python3 spec/validate.py`, `python3 spec/audit.py --selftest`, `python3 scripts/repo_gate.py` - clean. - The blocking CI prose run - clean. ## Correction: this branch carries the TODO bookkeeping too An earlier revision of this description said the `TODO.md` edits were held back to avoid conflicting with #595. **That was wrong.** This branch was cut from #595's branch rather than from `develop`, so it already contained that commit, and the description claimed an absence a reader could disprove from the diff. The review then caught the real consequence: the cluster entry still promised a bare drive letter after the implementation dropped that shape. Correcting the bullet was the smaller half of the answer, since this is the pull request that ships the cluster, so `c6fca9e` applies step 9 and deletes it rather than leaving a corrected description of finished work. The disproved claim is recorded here and in the commit message, alongside the measurement that disproved it. **This makes #595 redundant**, since every line of it is already in this branch. Closing it as superseded is the maintainer's call rather than something this branch does. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ptr727
added a commit
that referenced
this pull request
Aug 8, 2026
… that blocks on what it cannot read (#609) Promotes 14 commits from `develop`. Merge with a **merge commit** (`gh pr merge --merge`), never a squash, and **without `--delete-branch`**, since this pull request's head is `develop` itself. Closes #607 through the closing keyword already carried in `530cf71`, which is why it is not repeated here. ## The prose backlog, cleared end to end `#600`, `#604`, `#605`, `#606` took the tree from **557 findings across 45 files to 0 across 0**, in four batches ordered by surface: snippets, comments, hub-only Markdown, then the carried files. Each batch measured the checker's own exemption against the live corpus *before* sweeping, and twice the measured answer was **"do not change the checker"**, which is a result of that pass rather than a skipped one. `#594` added the floor that makes those numbers trustworthy: a diff-scoped run now asserts what it actually scanned, since a check whose scan matches nothing reports zero findings and reads exactly like a pass. One finding from that work is worth carrying up: an exemption that is too **loose** produces silence rather than false positives. #519 recorded the governance files as clean; today's checker reports 38 findings against those same files as they stood at the commit that measured them. ## A review loop that fails closed `#599`, `#601`, `#602`, `#603` and `#608` are one arc on `scripts/pr_review.py`, each removing a shape in which the loop reported a clean pass over a review it had misread: - **`#599`** removed the shape a reply kept failing in, by taking the thread's *words* rather than an id, so there is no argument a hand-typed `PRRT_...` fits in. - **`#602`** made `claims` resolve what a description points at rather than what it looks like. - **`#603`** gave a disproved claim a home the next round reads. - **`#608`** reads the file-coverage line, and then generalizes: every reader keys on a structural marker, so a marker that changes spelling is a section the reader stops finding and reports as absent. The digest now vets headings, `<summary>` texts, metadata labels, coverage wordings and the reviewer login against an inventory measured from **332 review bodies**, and **blocks on anything outside it**, exit `43`, with the remedy stated as filing an issue on the hub. Whether to merge regardless is the maintainer's decision. `GOVERNANCE.md` merge gate went from four preconditions to **five** accordingly. ## Governance and tooling - **`#593`** states which checkout an agent works in and what the hub is, which is the host-wide routing the repositories that most need it cannot carry. - **`#596`** gates the pattern-detectable half of the representative-data rule, honest that no pattern closes the name-shaped case. - **`#598`** declares where a repository states what CI cannot verify. - **`#592`** regrouped `TODO.md` by what ships rather than by what it touches, so a `###` heading is one pull request. - **`#601`** ended a `gh push` argument list at a newline rather than only at `&&`, fixing a write-guard over-block. ## Verification Run on `develop` at `530cf71` immediately before opening this: `test_pr_review.py` (174), `test_prose_lint.py`, `test_repo_gate.py`, `spec/audit.py --selftest`, `gh-write-guard.py --selftest`, `spec/validate.py`, `repo_gate.py`, the prose gate in both CI invocations, markdownlint and editorconfig-checker. All clean. ## Not carried by this promotion - **#519 is complete and still open.** `TODO.md` holds its closing evidence under "Verified Complete, Awaiting Close". Closing it is the maintainer's call, so no keyword for it appears here. - **The re-vendor debt is now nine files.** `#606` queued seven, and `#608` changed `GOVERNANCE.md` "PR Review Etiquette" (`verbatim`) and `.github/copilot-instructions.md` (`intent`) on top. The `intent` half produces no hash and therefore no audit finding, which is why the Fleet Sweeps entry names those files by hand. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Every verdict a
--diffrun reports rests on a scope the run never checked. A gate that finds nothing is indistinguishable from a gate with nothing to find, and the exit code is the same either way.Four routes to that false clean are already on record, each closed by a guard written after a reviewer noticed it: an unresolvable base widening to a whole-tree scan, a multi-line
pathsinput read only to its first newline, a diff taken in one repository while scanning another, and a path under no repository at all. Per-route guards are the wrong shape for a fifth, because the fifth is found by a reviewer or not at all.LEAST_PLAUSIBLEis already the floor on a whole-tree sweep, so the shape exists in the file and the--diffpath is what lacked it.Fresh evidence for the entry came from the session before this one, where
prose_lint --diff --summaryprinted0 violation(s) across 0 file(s)on a real change and distinguishing that from a scoping failure took a hand-written call intochanged_lines()to prove the scope was non-empty.What changed
A run that resolves a non-empty diff and matches none of the files it names now exits 2 naming them, instead of exiting 0.
Zero alone is not the test. A change touching only files the rules do not read, an image or a lock file, legitimately matches nothing, so the comparison is against the diff's own list of files this run could have read.
unread_diff_files()builds that list, andasked_about()keeps a caller who narrowed the scan on purpose from being told the narrowing is a defect.The exit code is 2 rather than 1, matching the two guards beside it, so a caller can tell "the gate could not run" from "the gate ran and found something".
The floor found the fifth route immediately
git diffreports repository-relative keys while discovery keys off the directory the run started in. Run from a subdirectory, the intersection is empty and the run reports clean. No existing guard covers it, and it is whyunread_diff_files()resolves keys against the repository top level: reading them against the working directory would leave the list empty in exactly the case it exists to catch.Measured on this branch's own diff, run from
scripts/, with the previous revision extracted to a temporary path so both run against the same tree. The error text is verbatim, wrapped here for width:Both directions are tested, and the second one is the larger half
Ten cases. Three fail against the previous revision, which is what proves the floor bites. The other seven pass against both revisions on purpose: they hold the direction that matters more, that a legitimate zero stays a clean run. A gate that cries wolf on a commit adding a logo is how a safety check stops being read.
The must-not-fire cases are a deliberate narrowing, an empty diff, an excluded file, a generated tree, a path the diff names that no longer exists, and a binary. Each was confirmed against a real run as well as a mocked one, the binary case with an actual PNG committed in a scratch repository.
Verification
python3 scripts/test_prose_lint.py- 173 pass, up from 163.python3 spec/validate.py,python3 spec/audit.py --selftest,python3 scripts/repo_gate.py- all clean.python3 scripts/prose_lint.py . --check charset --check dupword --check spelling- clean, the blocking CI run.python3 scripts/prose_lint.py . --diff develop- clean, and it caught two comment blocks in this change breaking the one-sentence-per-line rule, which are fixed.developat3f91868rather than only on the base the work started from.Honest limits
An absolute path argument produces an empty list rather than a finding, so the floor falls back to today's behavior there instead of over-blocking. Deletions never enter the scope map at all, since a deletion hunk yields an empty line range, so they cannot trigger it.
The CI jobs in this repo call
prose_lint.pywithout--diff, so nothing here changes what this repo's own pipeline reports. The consumer is.github/actions/prose-gate, which passes--diffand runs from the caller's repository root.Backlog
Carries the
TODO.mdcluster "The Prose Gate Scope Floor" whole. It unblocks "The Representative-Data Path Check", which is worth having only once the gate can prove it read something. Per selection step 9 the cluster is deleted when this merges, which is a follow-up commit rather than part of this diff.🤖 Generated with Claude Code