Declare where a repository states what CI cannot verify - #598
Merged
Conversation
A mandatory pre-pull-request gate lived in a repository's own topical doc under a heading of that repository's invention, and an agent routing by the doc's description never opened it. Nothing in the chain broke a rule, so it produced no signal: the description named the file by its most visible function, which was a post-merge one, and it was accurate. Fix the locating half rather than the wording half. "Verification Discipline" gains a bullet requiring a change's checks to be read from what the repository declares before any is run, since CI's coverage is not that list and green is the precise signal that the unreachable half was skipped. The bullet leads the section because the failure is upstream of every other bullet there, each of which assumes the gate is already known. Declare `Local Verification` as a sixth OPERATIONS.md heading so finding the check never depends on how well one repository worded a pointer. It leads the file as the only pre-merge heading. This repository's own OPERATIONS.md becomes the worked example, naming the live fleet audit and configure.sh check as the two verifications a pull request runner cannot perform. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR makes “local-only” verification steps discoverable and standardized across the fleet by declaring a required OPERATIONS.md heading (Local Verification) and adding governance that agents must locate required checks from repository declarations (not from CI’s green status).
Changes:
- Add a new first bullet to
GOVERNANCE.md“Verification Discipline” requiring agents to locate all checks a change owes (including non-CI checks) before running any. - Update the section model and standup guidance to require
OPERATIONS.mdto lead with aLocal Verificationheading (now 6 standardized headings total). - Update this repo’s
OPERATIONS.mdto include a worked-exampleLocal Verificationsection describing checks CI cannot run for this hub repo, and track the downstream sweep inTODO.md.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| TODO.md | Records the fleet sweep needed to propagate the new Local Verification heading and links it to #597. |
| STANDUP.md | Updates baseline guidance to require OPERATIONS.md to lead with Local Verification and expands the heading list to six. |
| spec/section-model.md | Declares Local Verification as the leading required OPERATIONS.md heading and explains why it must exist. |
| OPERATIONS.md | Adds a worked-example Local Verification section describing which checks CI cannot perform for this hub repo. |
| GOVERNANCE.md | Adds a first “Verification Discipline” bullet requiring explicit location of all required checks (CI is not the source of truth). |
The section told a reader to run `configure.sh check` from the repository root, where the script lives at `repo-config/configure.sh`, so following the instruction yields command not found. Give both commands the path and the prefix the runbooks below already use, and point at those runbooks for the argument lists rather than restating them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner
Author
|
From The count is 13, not twelve. Exact matches against the five declared headings: zero. Confirmed rather than eyeballed — One is a near-miss, and it is the part worth designing around. Blog has That matters for the heading check in #523's cluster in a way a from-scratch repository would not show:
The near-miss also suggests the failure mode the heading check will mostly find in existing repositories is synonym drift rather than absence. A repository that wrote its operational document before the spec declared the headings will have covered the same ground under its own names, so the useful output is a mapping from what the file has to what the spec wants, not a list of five things it lacks. Nothing here asks for a change to the decision in #597, which reads correctly from this end, including that the routing description was accurate and the reasoning from it was sound. Recording it because Blog is the worked example the sweep will hit first, and because |
ptr727
added a commit
that referenced
this pull request
Aug 8, 2026
… that blocks on what it cannot read (#609) Promotes 14 commits from `develop`. Merge with a **merge commit** (`gh pr merge --merge`), never a squash, and **without `--delete-branch`**, since this pull request's head is `develop` itself. Closes #607 through the closing keyword already carried in `530cf71`, which is why it is not repeated here. ## The prose backlog, cleared end to end `#600`, `#604`, `#605`, `#606` took the tree from **557 findings across 45 files to 0 across 0**, in four batches ordered by surface: snippets, comments, hub-only Markdown, then the carried files. Each batch measured the checker's own exemption against the live corpus *before* sweeping, and twice the measured answer was **"do not change the checker"**, which is a result of that pass rather than a skipped one. `#594` added the floor that makes those numbers trustworthy: a diff-scoped run now asserts what it actually scanned, since a check whose scan matches nothing reports zero findings and reads exactly like a pass. One finding from that work is worth carrying up: an exemption that is too **loose** produces silence rather than false positives. #519 recorded the governance files as clean; today's checker reports 38 findings against those same files as they stood at the commit that measured them. ## A review loop that fails closed `#599`, `#601`, `#602`, `#603` and `#608` are one arc on `scripts/pr_review.py`, each removing a shape in which the loop reported a clean pass over a review it had misread: - **`#599`** removed the shape a reply kept failing in, by taking the thread's *words* rather than an id, so there is no argument a hand-typed `PRRT_...` fits in. - **`#602`** made `claims` resolve what a description points at rather than what it looks like. - **`#603`** gave a disproved claim a home the next round reads. - **`#608`** reads the file-coverage line, and then generalizes: every reader keys on a structural marker, so a marker that changes spelling is a section the reader stops finding and reports as absent. The digest now vets headings, `<summary>` texts, metadata labels, coverage wordings and the reviewer login against an inventory measured from **332 review bodies**, and **blocks on anything outside it**, exit `43`, with the remedy stated as filing an issue on the hub. Whether to merge regardless is the maintainer's decision. `GOVERNANCE.md` merge gate went from four preconditions to **five** accordingly. ## Governance and tooling - **`#593`** states which checkout an agent works in and what the hub is, which is the host-wide routing the repositories that most need it cannot carry. - **`#596`** gates the pattern-detectable half of the representative-data rule, honest that no pattern closes the name-shaped case. - **`#598`** declares where a repository states what CI cannot verify. - **`#592`** regrouped `TODO.md` by what ships rather than by what it touches, so a `###` heading is one pull request. - **`#601`** ended a `gh push` argument list at a newline rather than only at `&&`, fixing a write-guard over-block. ## Verification Run on `develop` at `530cf71` immediately before opening this: `test_pr_review.py` (174), `test_prose_lint.py`, `test_repo_gate.py`, `spec/audit.py --selftest`, `gh-write-guard.py --selftest`, `spec/validate.py`, `repo_gate.py`, the prose gate in both CI invocations, markdownlint and editorconfig-checker. All clean. ## Not carried by this promotion - **#519 is complete and still open.** `TODO.md` holds its closing evidence under "Verified Complete, Awaiting Close". Closing it is the maintainer's call, so no keyword for it appears here. - **The re-vendor debt is now nine files.** `#606` queued seven, and `#608` changed `GOVERNANCE.md` "PR Review Etiquette" (`verbatim`) and `.github/copilot-instructions.md` (`intent`) on top. The `intent` half produces no hash and therefore no audit finding, which is why the Fleet Sweeps entry names those files by hand. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Answers #597, filed from a downstream repository after an agent skipped a mandatory pre-pull-request gate while following every carried rule correctly.
The failure
A repository's
OPERATIONS.mdheld a verification step CI structurally cannot perform, under a heading of that repository's own invention. The routing sentence in itsAGENTS.mdnamed the file by its most visible function, which was a post-merge one. An agent deciding how to verify a change before opening a pull request therefore had no reason to open the file, and skipped the one check that could have caught the regression. Nothing in that chain broke a rule, which is why it produced no signal.Why the existing rules do not reach it
"Verification Discipline" is the natural home and does not cover it. Every bullet there assumes the agent already knows which gate to run: a gate that stops gating, a filter that matches less, a green check that is not evidence. This failure is upstream of all of them, because the gate was never located, so there was no green to distrust.
What lands
The locating half, not the wording half. Fixing the one repository's routing sentence leaves the next repository to reinvent the same shape, so the fix is a carried rule that points at a fleet-declared location.
GOVERNANCE.md"Verification Discipline" gains a bullet, placed first because the failure is upstream of the rest: a change's checks are read from what the repository declares before any is run, CI's coverage is not that list, and green is the precise signal that the unreachable half was skipped. It names why a document's own description cannot be relied on to find the check.spec/section-model.mdandSTANDUP.mddeclareLocal Verificationas a sixthOPERATIONS.mdheading, leading the file as the only pre-merge one. Both restatements of "those five headings" move with it.OPERATIONS.mdbecomes the worked example. The gates block moves under the new heading, and the section states the two verifications a pull request runner here cannot perform, the live fleet audit andconfigure.sh check, each of which reads another repository over the API.spec/audit.py --selftestis all CI runs of the audit, against fixtures.TODO.mdindexes A verification gate CI cannot run is discoverable only through a repo's own description of its topical doc #597 under "Fleet Sweeps" rather than as complete, since every repository still owes the heading, riding the pending re-vendor visit.The gap this leaves, stated rather than hidden
The audit reports nothing here today, because
OPERATIONS.mdis presence-checked only, so a repository using none of the declared headings passes. The heading check is #523's cluster, and until it ships the sweep is verified by reading each file rather than by a run.TODO.mdrecords that.Verification
Every gate in the runbook: the four self-test suites,
repo_gate.py,spec/validate.py, bothprose_lint.pyinvocations,editorconfig-checker, and markdownlint over 44 files with 0 issues. The live audit runs clean at 0 defect findings, its drift output being the already-recorded re-vendor backlog.One note on the run itself, since this change is about false cleans.
prose_lint.py . --diff origin/developreported 0 violations across 0 files, because nothing was committed at that point, so the warn set was re-run against the five paths explicitly. It reports 14semicolonfindings inGOVERNANCE.md, identical to the count on the same file with the change stashed, so this adds none.Review round: an unrunnable command, in the section about unrunnable checks
Round one flagged that
configure.sh checkis not invokable from the repository root, since the script lives atrepo-config/configure.sh, so a reader following the instruction getscommand not found. Valid, and pointed at a section whose whole subject is a check CI cannot run.Fixed in
b9f9640, and swept as a class rather than as an instance. This branch adds three command references, and a second carried the same defect:spec/audit.py --selftestwas missing thepython3prefix CI actually invokes it with. Both now carry the path and the prefix the runbooks below use, and the sentence points at those runbooks for the argument lists rather than restating them. Round two againstb9f9640generated no new comments.Worth recording from the loop itself: the
gh-write-guardhook blocked the resolve mutation because the thread id was hand-typed from the previous query's output rather than re-captured live. That is the rule catching a real instance rather than a hypothetical one, and the mutation succeeded once the id came from a query in the same invocation.🤖 Generated with Claude Code