AGENTS.md: add a verification-discipline section

[ptr727]

Proposal

Add a Verification Discipline section to AGENTS.md covering the checks that separate work actually done from work that merely reports success. Implemented and in use at ptr727/ESPHome-NonRoot#165; raising it here because none of it is repo-specific.

The unifying property: each failure below is green. A skipped job and a passing job are indistinguishable in the aggregator; a pattern that matches less still exits zero; a gate that stops gating still reports success. Nothing in the existing lint, review, or status-check layers catches any of them.

Where it fits against what already exists

Three adjacent rules already cover narrower cases, so this complements rather than duplicates them:

• "Running the Linters Locally" documents how to invoke each linter, not the rule that all of them run before every push.
• The changes-job note ("must succeed, not just not-fail... silently skipped") is the aggregator-specific instance of the general "green is not evidence" rule.
• The line-endings section covers preserving endings and notes file is unreliable, but not the regex mechanism that corrupts them.

Proposed rules

New, not covered anywhere today:

1. A test must assert the mechanism it names. Label each case by the behavior it proves, and satisfy yourself it would fail if that mechanism broke. A case that passes for an incidental reason - the right answer reached by the wrong path - is worse than no case, because it is later cited as evidence. (This one caused a real defect: a gate test passed because the branch happened to contain a matching file, not because the fail-open path worked. It was cited twice as proof before review found the gap.)
2. Gates, filters, and gate-like watchers must fail loud, never narrow quietly. A pattern that silently matches less, an allowlist that silently stops matching, or a gate that silently stops gating all report success while doing nothing. When a construct exists to notice something, make the not-noticing case produce an error or an annotation.
3. Run the repo's whole lint gate before every push, not the parts that look relevant. CI runs all of them, so a partial local run only defers the failure - and the tool most likely to catch a given change is often the one it seems least about (an edit that manipulates line endings is exactly when editorconfig-checker matters).
4. Editing CRLF files programmatically: . matches \r in a regex, so a captured line keeps its carriage return and rejoining with \r\n yields CRCRLF. Prefer line-based edits (splitlines(keepends=True)) or literal replacement over regex reassembly. This is the mechanism behind the existing "a programmatic edit can silently flip endings" warning, and it is worth naming because the corruption is invisible in a diff.

Generalizations of rules that exist in narrower form:

5. A green check is not evidence the work happened. A skipped job and a passing job are indistinguishable in the aggregated required check. When a job exists to exercise something, confirm from its log that it ran and produced the output it promises. (Generalizes the changes-job note beyond that one job.)
6. A workflow change is only fully exercised by CI. Extracting a run: block and executing it locally validates the script and nothing else - secrets: inherit, permissions:, needs: wiring, and reusable-workflow inputs are only resolved by a real run. (Complements the existing permissions-before-if: and success/skipped allowlist notes, which are instances of the same class.)

Also worth considering

The README Rules index could carry one or two of these under Always - "Run the full lint gate before pushing" and "Make gates fail loud, never narrow quietly" are the two that read as fleet-wide invariants rather than technique. That would need the Table of Contents updated to match.

Reference implementation

ptr727/ESPHome-NonRoot#165 has the wording as merged, if it is easier to adapt than to rewrite.

[ptr727]

Amendment: a seventh rule, and a placement note

Since filing this I hit the same defect three more times in one change set, and it is not covered by the six rules above. Those are about verifying that work happened; this one is about documentation that was true and silently became false.

The pattern

Adding a change-gate to a CI job invalidated three separate pieces of prose describing the previous always-on/always-off behavior:

• a Mermaid diagram label in the flow section
• a D-numbered guarantee in the behavioral contract
• a reusable workflow's input comment, which is the contract its callers read

Every one was accurate when written. The guarantees I was consciously editing got updated; the incidental prose elsewhere did not. Two were caught by review, one only by sweeping after the second.

No linter detects this. markdownlint, cspell, actionlint, and editorconfig-checker all pass on a comment that is merely untrue - it is well-formed, correctly spelled, and wrong.

Proposed rule

When you change a behavior, search for prose that asserts the old one. Updating the guarantee or rule you are consciously editing is not enough: comments, diagram labels, reusable-workflow input descriptions, and audit statements elsewhere may still describe the prior behavior. Grep for the old behavior's distinctive phrasing and fix every instance. No linter can catch a claim that is merely untrue, so this sweep is the only mechanism that will.

And a companion worth stating explicitly, because it changes how a review is answered:

A review flags an instance; fix the class. When a reviewer cites one stale claim, one silent-narrowing pattern, or one mis-worded contract, sweep for its siblings before replying. Reviewers sample; they do not enumerate.

The companion is not hypothetical: in this case the review flagged the input comment, and the sweep found a second instance in the contract that it had not seen.

Placement note

This one probably belongs under Documentation Style Conventions rather than Verification Discipline. That section already carries "write docs in the current state, not as a change from a prior one", which governs how to phrase a doc. This is the maintenance counterpart - how to keep it current when the behavior underneath it moves - so the two sit naturally together.

The other six from the original proposal still read as Verification Discipline.

[ptr727]

Landed in #358 (merged to develop), including the amendment.

## Verification Discipline sits between Documentation Style Conventions and PR Review Etiquette - write it right, verify it, then review - and opens with the unifying property you identified: every failure in it is green, so no linter, status check, or review layer catches any of them. It carries the six proposed rules plus the amendment's review companion ("a review flags an instance; fix the class").

Rule 7 - search for prose asserting the old behavior after changing it - went under Documentation Style Conventions per your placement note, beside the present-tense rule as its maintenance counterpart: that rule governs how to phrase a doc, this one how to keep it true when the behavior underneath moves.

One placement judgment to flag: you grouped the review companion with rule 7 but did not place it. I put it in Verification Discipline rather than the docs section, since it generalizes past stale prose to any flagged defect class - a silent-narrowing pattern, a mis-worded contract. Say the word if you'd rather keep the pair together.

On complementing rather than duplicating: each generalization now cross-references its narrower instance instead of restating it - fail-loud to WORKFLOW.md D8.4 (landed via #354), the lint-gate rule to "Running the Linters Locally", green-check to the changes-job note, and the CRLF rule to the Line Endings warning whose mechanism it names. I also folded the mirror failure into rule 4: a text-mode rewrite silently flattens CRLF to LF, the inverse of the CRCRLF case - I hit exactly that two days ago, and CI caught it rather than my local run, which is rule 3 in action.

README Rules -> Always gained the two that read as fleet-wide invariants: run the whole lint gate before pushing, and make gates fail loud. Both are bullets under an existing heading, so the auto-generated Table of Contents needed no change.

Copilot review: clean pass, no findings.