Skip to content

Assert a floor on what a diff-scoped prose run actually scanned - #594

Merged
ptr727 merged 1 commit into
developfrom
feat/prose-gate-scope-floor
Aug 7, 2026
Merged

Assert a floor on what a diff-scoped prose run actually scanned#594
ptr727 merged 1 commit into
developfrom
feat/prose-gate-scope-floor

Conversation

@ptr727

@ptr727 ptr727 commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Why

Every verdict a --diff run reports rests on a scope the run never checked. A gate that finds nothing is indistinguishable from a gate with nothing to find, and the exit code is the same either way.

Four routes to that false clean are already on record, each closed by a guard written after a reviewer noticed it: an unresolvable base widening to a whole-tree scan, a multi-line paths input read only to its first newline, a diff taken in one repository while scanning another, and a path under no repository at all. Per-route guards are the wrong shape for a fifth, because the fifth is found by a reviewer or not at all. LEAST_PLAUSIBLE is already the floor on a whole-tree sweep, so the shape exists in the file and the --diff path is what lacked it.

Fresh evidence for the entry came from the session before this one, where prose_lint --diff --summary printed 0 violation(s) across 0 file(s) on a real change and distinguishing that from a scoping failure took a hand-written call into changed_lines() to prove the scope was non-empty.

What changed

A run that resolves a non-empty diff and matches none of the files it names now exits 2 naming them, instead of exiting 0.

Zero alone is not the test. A change touching only files the rules do not read, an image or a lock file, legitimately matches nothing, so the comparison is against the diff's own list of files this run could have read. unread_diff_files() builds that list, and asked_about() keeps a caller who narrowed the scan on purpose from being told the narrowing is a defect.

The exit code is 2 rather than 1, matching the two guards beside it, so a caller can tell "the gate could not run" from "the gate ran and found something".

The floor found the fifth route immediately

git diff reports repository-relative keys while discovery keys off the directory the run started in. Run from a subdirectory, the intersection is empty and the run reports clean. No existing guard covers it, and it is why unread_diff_files() resolves keys against the repository top level: reading them against the working directory would leave the list empty in exactly the case it exists to catch.

Measured on this branch's own diff, run from scripts/, with the previous revision extracted to a temporary path so both run against the same tree. The error text is verbatim, wrapped here for width:

$ git show HEAD~1:scripts/prose_lint.py > /tmp/prev_prose_lint.py
$ cd scripts && python3 /tmp/prev_prose_lint.py . --diff develop --check dupword
$ echo $?
0

$ cd scripts && python3 prose_lint.py . --diff develop --check dupword
error: the diff against 'develop' names 2 readable file(s) this run was asked about, and the
scan matched none of them: scripts/prose_lint.py, scripts/test_prose_lint.py. Refusing to
report a clean run, since a gate that read nothing is indistinguishable from a gate with
nothing to read. Check that the run starts at the repository top level and that the requested
paths cover the change.
$ echo $?
2

Both directions are tested, and the second one is the larger half

Ten cases. Three fail against the previous revision, which is what proves the floor bites. The other seven pass against both revisions on purpose: they hold the direction that matters more, that a legitimate zero stays a clean run. A gate that cries wolf on a commit adding a logo is how a safety check stops being read.

The must-not-fire cases are a deliberate narrowing, an empty diff, an excluded file, a generated tree, a path the diff names that no longer exists, and a binary. Each was confirmed against a real run as well as a mocked one, the binary case with an actual PNG committed in a scratch repository.

Verification

  • python3 scripts/test_prose_lint.py - 173 pass, up from 163.
  • python3 spec/validate.py, python3 spec/audit.py --selftest, python3 scripts/repo_gate.py - all clean.
  • python3 scripts/prose_lint.py . --check charset --check dupword --check spelling - clean, the blocking CI run.
  • python3 scripts/prose_lint.py . --diff develop - clean, and it caught two comment blocks in this change breaking the one-sentence-per-line rule, which are fixed.
  • Re-run after rebasing onto develop at 3f91868 rather than only on the base the work started from.

Honest limits

An absolute path argument produces an empty list rather than a finding, so the floor falls back to today's behavior there instead of over-blocking. Deletions never enter the scope map at all, since a deletion hunk yields an empty line range, so they cannot trigger it.

The CI jobs in this repo call prose_lint.py without --diff, so nothing here changes what this repo's own pipeline reports. The consumer is .github/actions/prose-gate, which passes --diff and runs from the caller's repository root.

Backlog

Carries the TODO.md cluster "The Prose Gate Scope Floor" whole. It unblocks "The Representative-Data Path Check", which is worth having only once the gate can prove it read something. Per selection step 9 the cluster is deleted when this merges, which is a follow-up commit rather than part of this diff.

🤖 Generated with Claude Code

A gate that finds nothing is indistinguishable from a gate with nothing to
find, and every verdict a --diff run reports rests on a scope it never
checked. Four routes to the same false clean are already on record, each
closed by a guard written after a reviewer noticed it: an unresolvable base
widening to a whole-tree scan, a multi-line paths input read only to its
first newline, a diff taken in one repository while scanning another, and a
path under no repository at all. The fifth route gets found the same way or
not at all, which is why this asserts a floor rather than adding a guard.

A run that resolves a non-empty diff and matches none of the files it names
now exits 2 naming them, rather than exiting 0. Zero alone is not the test,
since a change touching only files the rules do not read, an image or a lock
file, legitimately matches nothing, so the comparison is against the diff's
own list of files this run could have read.

The floor found the fifth route immediately. git diff reports
repository-relative keys while discovery keys off the directory the run
started in, so running the gate from a subdirectory intersects to nothing and
reported clean. unread_diff_files resolves against the repository top level
for that reason: reading the diff's paths against the working directory would
leave the list empty in exactly the case it exists to catch.

Ten cases lock it. Three fail against the previous revision, and seven pass
against both because they hold the other direction, that a deliberate
narrowing, an empty diff, an excluded file, a generated tree, a deletion and
a binary are each a clean run rather than a scoping failure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 7, 2026 00:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR strengthens the reliability of scripts/prose_lint.py --diff by asserting a minimum “diff scope floor”: if a non-empty diff is resolved but the scan matches none of the diff’s readable, in-scope files, the run now fails (exit code 2) rather than reporting a false clean.

Changes:

  • Add asked_about() and unread_diff_files() helpers to determine which diff-named files the run was asked to cover and could have read.
  • Update main() to exit 2 (with a targeted error message) when a non-empty diff yields zero matched files but the diff names readable files that should have been scanned.
  • Add a new TestDiffScopeFloor suite covering both “must fire” and “must not fire” scenarios (including the subdirectory-run key mismatch).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
scripts/prose_lint.py Implements the diff-scope floor check and supporting helpers, returning exit code 2 on confirmed scoping failures.
scripts/test_prose_lint.py Adds focused unit tests validating the new floor behavior and guarding legitimate zero-match cases.

@ptr727
ptr727 merged commit c8687c5 into develop Aug 7, 2026
7 checks passed
@ptr727
ptr727 deleted the feat/prose-gate-scope-floor branch August 7, 2026 00:49
ptr727 added a commit that referenced this pull request Aug 7, 2026
## Why

`GOVERNANCE.md` "Representative Data in Agent-Authored Text" says
agent-authored text illustrates with constructed data rather than data
observed in the maintainer's environment, and says in the same breath
that no pattern closes it. This gates the half a pattern can reach.

It is introduced as that half deliberately. The leak that motivated the
rule was name-shaped and sat in a pull request comment, which no
committed-file linter reads, so a check mistaken for the answer is what
stops anyone looking at the exposure the rule actually exists for.

Carries the `TODO.md` cluster "The Representative-Data Path Check",
which #595 released from `blocked` once the scope floor shipped in #594.

## The shapes were measured, not assumed

The backlog recorded four shapes, `/home/<name>`, `/Users/<name>`,
`C:\Users\<name>`, and a bare drive letter. Run against this repository,
**a bare drive letter matched 11 files and named a path in none of
them.**

An escaped newline after any word ending in a letter and a colon reads
as a drive letter, so a YAML fixture is full of them:

| Matched text | What it actually is |
| --- | --- |
| `e:\n    needs` | `jobs:\n    needs` inside a test fixture string |
| `s:\n  a` | `jobs:\n  a:` inside a test fixture string |
| `C:\tmp` (8 hits) | a temp path in fixtures for the string-literal
rule |

That shape is dropped. The one kept is a drive letter followed by
`Users`, which matches nothing in the tree today. This is narrower than
the committed prose, which says "an absolute home path or a drive
letter", so the narrowing is stated here rather than left for a reader
to discover.

## The exemption is built, not listed

A wrong exemption hands out a work list that damages correct documents,
and this rule has two self-referential problems that a file-name
allowlist would solve badly and briefly.

**The rule's own wording quotes the shapes it forbids.** Requiring a
real user segment solves it by construction: the documentation
placeholder is not a valid account name, so it describes the shape
without matching it. No exemption names `GOVERNANCE.md`, and none goes
stale when the text moves.

| Probe | Verdict |
| --- | --- |
| `C:\Users\<name>` in documentation | no match |
| `/home/<name>` in documentation | no match |
| `/home/vscode/.ssh` | exempt, container account |
| a real absolute home path | **flagged** |

**Container accounts are exempt by name.** Every home path in this
repository, all 15, is `/home/vscode` from the devcontainer snippets and
the doc describing them. Without that exemption the rule would open with
15 findings and no true positive among them.

**The gate reads its own source**, so the bait paths in the tests are
assembled from parts and a case asserts that neither `prose_lint.py` nor
its test file carries a literal. That is the approach the file already
takes for non-ASCII, which it writes as escapes for exactly this reason.

## Operational repositories are exempt, and told so

An operational repository's runbook carries the literal path an operator
types. That is the repository's own content rather than an agent quoting
an environment it observed, which is the distinction the rule is about.

The model is read from what the repository carries rather than from the
hub registry, which a downstream checkout cannot see: `spec/files.json`
declares `repo-config/operational/develop.json` for the operational
model and `repo-config/develop.json` for the release one. The hub
carries both, being the template for each, so it reads as release rather
than exempting itself from a rule it authors.

**The skip is announced on stderr**, because a rule that silently stops
running reads as a rule that passed, which is the same failure #594
closed on the other side.

## Scope note

Unlike every other prose rule, this one reads fenced blocks and
configuration values rather than stripped prose. A transcript pasted
from a terminal and a bind mount naming a real home are the exposure in
its most consequential forms, and neither is a sentence.

## Verification

- `python3 scripts/test_prose_lint.py` - 188 pass, up from 173. Of the
14 new cases, 10 fail against `develop` and 4 pass against both, holding
the must-not-fire direction.
- `python3 scripts/prose_lint.py . --check home-path` - clean across the
tracked tree.
- `python3 scripts/prose_lint.py . --summary` - 557 violations across 45
files, the #519 backlog figure unchanged, with `home-path` contributing
none. Adding a rule to `DEFAULT_RULES` did not move the number.
- `python3 spec/validate.py`, `python3 spec/audit.py --selftest`,
`python3 scripts/repo_gate.py` - clean.
- The blocking CI prose run - clean.

## Correction: this branch carries the TODO bookkeeping too

An earlier revision of this description said the `TODO.md` edits were
held back to avoid conflicting with #595. **That was wrong.** This
branch was cut from #595's branch rather than from `develop`, so it
already contained that commit, and the description claimed an absence a
reader could disprove from the diff.

The review then caught the real consequence: the cluster entry still
promised a bare drive letter after the implementation dropped that
shape. Correcting the bullet was the smaller half of the answer, since
this is the pull request that ships the cluster, so `c6fca9e` applies
step 9 and deletes it rather than leaving a corrected description of
finished work. The disproved claim is recorded here and in the commit
message, alongside the measurement that disproved it.

**This makes #595 redundant**, since every line of it is already in this
branch. Closing it as superseded is the maintainer's call rather than
something this branch does.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ptr727 added a commit that referenced this pull request Aug 8, 2026
… that blocks on what it cannot read (#609)

Promotes 14 commits from `develop`. Merge with a **merge commit** (`gh
pr merge --merge`), never a squash, and **without `--delete-branch`**,
since this pull request's head is `develop` itself.

Closes #607 through the closing keyword already carried in `530cf71`,
which is why it is not repeated here.

## The prose backlog, cleared end to end

`#600`, `#604`, `#605`, `#606` took the tree from **557 findings across
45 files to 0 across 0**, in four batches ordered by surface: snippets,
comments, hub-only Markdown, then the carried files. Each batch measured
the checker's own exemption against the live corpus *before* sweeping,
and twice the measured answer was **"do not change the checker"**, which
is a result of that pass rather than a skipped one.

`#594` added the floor that makes those numbers trustworthy: a
diff-scoped run now asserts what it actually scanned, since a check
whose scan matches nothing reports zero findings and reads exactly like
a pass.

One finding from that work is worth carrying up: an exemption that is
too **loose** produces silence rather than false positives. #519
recorded the governance files as clean; today's checker reports 38
findings against those same files as they stood at the commit that
measured them.

## A review loop that fails closed

`#599`, `#601`, `#602`, `#603` and `#608` are one arc on
`scripts/pr_review.py`, each removing a shape in which the loop reported
a clean pass over a review it had misread:

- **`#599`** removed the shape a reply kept failing in, by taking the
thread's *words* rather than an id, so there is no argument a hand-typed
`PRRT_...` fits in.
- **`#602`** made `claims` resolve what a description points at rather
than what it looks like.
- **`#603`** gave a disproved claim a home the next round reads.
- **`#608`** reads the file-coverage line, and then generalizes: every
reader keys on a structural marker, so a marker that changes spelling is
a section the reader stops finding and reports as absent. The digest now
vets headings, `<summary>` texts, metadata labels, coverage wordings and
the reviewer login against an inventory measured from **332 review
bodies**, and **blocks on anything outside it**, exit `43`, with the
remedy stated as filing an issue on the hub. Whether to merge regardless
is the maintainer's decision.

`GOVERNANCE.md` merge gate went from four preconditions to **five**
accordingly.

## Governance and tooling

- **`#593`** states which checkout an agent works in and what the hub
is, which is the host-wide routing the repositories that most need it
cannot carry.
- **`#596`** gates the pattern-detectable half of the
representative-data rule, honest that no pattern closes the name-shaped
case.
- **`#598`** declares where a repository states what CI cannot verify.
- **`#592`** regrouped `TODO.md` by what ships rather than by what it
touches, so a `###` heading is one pull request.
- **`#601`** ended a `gh push` argument list at a newline rather than
only at `&&`, fixing a write-guard over-block.

## Verification

Run on `develop` at `530cf71` immediately before opening this:
`test_pr_review.py` (174), `test_prose_lint.py`, `test_repo_gate.py`,
`spec/audit.py --selftest`, `gh-write-guard.py --selftest`,
`spec/validate.py`, `repo_gate.py`, the prose gate in both CI
invocations, markdownlint and editorconfig-checker. All clean.

## Not carried by this promotion

- **#519 is complete and still open.** `TODO.md` holds its closing
evidence under "Verified Complete, Awaiting Close". Closing it is the
maintainer's call, so no keyword for it appears here.
- **The re-vendor debt is now nine files.** `#606` queued seven, and
`#608` changed `GOVERNANCE.md` "PR Review Etiquette" (`verbatim`) and
`.github/copilot-instructions.md` (`intent`) on top. The `intent` half
produces no hash and therefore no audit finding, which is why the Fleet
Sweeps entry names those files by hand.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants