test(docs): a requirement keyword outside the spec must name its source - #175
Merged
imran-siddique merged 1 commit intoAug 14, 2026
Conversation
Contributor
|
❔ Contributor Check: UNKNOWN
Automated check by AgenTrust Contributor Check. |
CONTRIBUTING.md says informative text "carries no RFC 2119 keywords and binds no implementation", and nothing checks it. Guides and crosswalks do need to state requirements, so the rule that holds mechanically is the weaker one: a requirement keyword under docs/ must be attributable to a document that binds. Run over main, the guard finds one line: docs/schema.md:66 states a SHOULD that no specification contains. Lowercased here. The three other keyword-bearing lines under docs/ all quote a source (Acta s2.2, s3.1, and the spec's §3.2.1) and pass unchanged, so the permissive half is exercised by real text rather than only by fixtures. The corpus is recovered from the tree, so a new directory under docs/ is covered the day it appears. Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
lywinged
force-pushed
the
test/normative-keywords-outside-spec
branch
from
August 14, 2026 08:29
584c6ab to
0a9a67a
Compare
imran-siddique
approved these changes
Aug 14, 2026
imran-siddique
left a comment
Member
There was a problem hiding this comment.
The guard scans the discovered docs corpus, exercises both reject and permit paths, excludes fenced examples, and lowers the one currently unattributed keyword. Its scope is attribution rather than citation validity, which matches the stated contract.
9 tasks
lywinged
added a commit
to lywinged/trace-spec
that referenced
this pull request
Aug 24, 2026
Four days of upstream and every one of these broke loudly rather than silently, which is the design working. Recorded here because what broke is more useful than the repair. The rebase skipped d365f8d, whose guard had landed upstream as agentrust-io#171. The commit also carried the gen_vectors.py fix that upstream did not take, so skipping it dropped that half and the generator drifted from its fixtures again. Restored. A commit that bundles two changes cannot be skipped by halves. The depth loader named surface/builder_chain/dependency_chain, which upstream renamed to surface/builder/transitive. It now reads the depth names from a fixture instead of restating them, so the next rename changes nothing here. The same loader read only `outcome`, and upstream moved the signal: a vector that cannot be established at a depth now reports accept with `verified_depth` short of it and the reason under `unresolved`, which separates cannot-establish from failed and is the better shape. Reading the verdict alone scored 04 and 06 as separating nothing. Measured properly the set is healthy: builder has two vectors, transitive three, plus the accepting control. docs/normative-crosswalk.md gains nine rows for spec 3.3.1, and its prose no longer states requirements it does not cite. That guard is upstream's, contributed from here as agentrust-io#175, and it was failing on this fork's own document. 603 pass; the three failures are the schema-const collision that predates this. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CONTRIBUTING.mdsays informative text "carries no RFC 2119 keywords andbinds no implementation". Nothing checks it, and the literal check would be
the wrong one - a crosswalk that cannot quote the requirement it maps is
useless. The rule that holds mechanically is weaker and still worth having:
a requirement keyword under
docs/must be attributable to a document thatbinds.
Run over main, the guard finds exactly one line:
No specification contains that requirement. Lowercased here.
The three other keyword-bearing lines under
docs/all name a source -Acta s2.2 and s3.1 in the Acta crosswalk, the spec's §3.2.1 in
verification.md - and pass unchanged. That matters more than the catch: the
permissive half of the rule is exercised by real text rather than only by
fixtures, so a guard that silently permitted everything would not look like
this.
Both failure directions are quiet. A doc that invents a requirement ships,
and the first symptom is an implementer citing something reviewers cannot
find in the spec. A doc that quotes a real requirement without saying where
from reads identically and cannot be checked either.
The corpus is recovered from the tree rather than listed in the test, so a
new directory under
docs/is covered the day it appears.Twelve tests. Fenced blocks are excluded - sample output and JSON carrying
a
MUSTstring are illustration, and scanning them would push authors toreword examples to satisfy the guard, which is worse than not having it.
One thing this will do
Applied to #162 as it stands, it flags line 280 and nothing else in that
file's 414 lines. That is not a defect report on #162: the keyword entered
in 1bcba06, during the review round, so it may well be deliberate. It
surfaces a genuine either/or for the maintainer - the RFC declares itself
informative at line 394, and one of the two statements gives. Raised on the
thread rather than left for CI to deliver.
Verification
388 passed, 1 skipped.
ruffclean. Reverting thedocs/schema.mdchangefails the guard, so the fix and the check are load-bearing on each other.