Skip to content

test(docs): a requirement keyword outside the spec must name its source - #175

Merged
imran-siddique merged 1 commit into
agentrust-io:mainfrom
lywinged:test/normative-keywords-outside-spec
Aug 14, 2026
Merged

test(docs): a requirement keyword outside the spec must name its source#175
imran-siddique merged 1 commit into
agentrust-io:mainfrom
lywinged:test/normative-keywords-outside-spec

Conversation

@lywinged

Copy link
Copy Markdown
Collaborator

CONTRIBUTING.md says informative text "carries no RFC 2119 keywords and
binds no implementation". Nothing checks it, and the literal check would be
the wrong one - a crosswalk that cannot quote the requirement it maps is
useless. The rule that holds mechanically is weaker and still worth having:
a requirement keyword under docs/ must be attributable to a document that
binds.

Run over main, the guard finds exactly one line:

docs/schema.md:66
  Custom values are allowed and SHOULD follow your organization's
  data classification policy.

No specification contains that requirement. Lowercased here.

The three other keyword-bearing lines under docs/ all name a source -
Acta s2.2 and s3.1 in the Acta crosswalk, the spec's §3.2.1 in
verification.md - and pass unchanged. That matters more than the catch: the
permissive half of the rule is exercised by real text rather than only by
fixtures, so a guard that silently permitted everything would not look like
this.

Both failure directions are quiet. A doc that invents a requirement ships,
and the first symptom is an implementer citing something reviewers cannot
find in the spec. A doc that quotes a real requirement without saying where
from reads identically and cannot be checked either.

The corpus is recovered from the tree rather than listed in the test, so a
new directory under docs/ is covered the day it appears.

Twelve tests. Fenced blocks are excluded - sample output and JSON carrying
a MUST string are illustration, and scanning them would push authors to
reword examples to satisfy the guard, which is worse than not having it.

One thing this will do

Applied to #162 as it stands, it flags line 280 and nothing else in that
file's 414 lines. That is not a defect report on #162: the keyword entered
in 1bcba06, during the review round, so it may well be deliberate. It
surfaces a genuine either/or for the maintainer - the RFC declares itself
informative at line 394, and one of the two statements gives. Raised on the
thread rather than left for CI to deliver.

Verification

388 passed, 1 skipped. ruff clean. Reverting the docs/schema.md change
fails the guard, so the fix and the check are load-bearing on each other.

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Contributor Check: UNKNOWN

Check Result
Profile UNKNOWN
Credential LOW
Overall UNKNOWN

Automated check by AgenTrust Contributor Check.

@github-actions github-actions Bot added the needs-review:UNKNOWN Contributor check flagged UNKNOWN risk label Aug 14, 2026
CONTRIBUTING.md says informative text "carries no RFC 2119 keywords and binds
no implementation", and nothing checks it. Guides and crosswalks do need to
state requirements, so the rule that holds mechanically is the weaker one: a
requirement keyword under docs/ must be attributable to a document that binds.

Run over main, the guard finds one line: docs/schema.md:66 states a SHOULD that
no specification contains. Lowercased here. The three other keyword-bearing
lines under docs/ all quote a source (Acta s2.2, s3.1, and the spec's §3.2.1)
and pass unchanged, so the permissive half is exercised by real text rather
than only by fixtures.

The corpus is recovered from the tree, so a new directory under docs/ is
covered the day it appears.

Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
@lywinged
lywinged force-pushed the test/normative-keywords-outside-spec branch from 584c6ab to 0a9a67a Compare August 14, 2026 08:29

@imran-siddique imran-siddique left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The guard scans the discovered docs corpus, exercises both reject and permit paths, excludes fenced examples, and lowers the one currently unattributed keyword. Its scope is attribution rather than citation validity, which matches the stated contract.

@imran-siddique
imran-siddique merged commit 62bcf32 into agentrust-io:main Aug 14, 2026
5 of 6 checks passed
@lywinged
lywinged deleted the test/normative-keywords-outside-spec branch August 18, 2026 05:01
lywinged added a commit to lywinged/trace-spec that referenced this pull request Aug 24, 2026
Four days of upstream and every one of these broke loudly rather than silently,
which is the design working. Recorded here because what broke is more useful than
the repair.

The rebase skipped d365f8d, whose guard had landed upstream as agentrust-io#171. The commit
also carried the gen_vectors.py fix that upstream did not take, so skipping it
dropped that half and the generator drifted from its fixtures again. Restored.
A commit that bundles two changes cannot be skipped by halves.

The depth loader named surface/builder_chain/dependency_chain, which upstream
renamed to surface/builder/transitive. It now reads the depth names from a fixture
instead of restating them, so the next rename changes nothing here.

The same loader read only `outcome`, and upstream moved the signal: a vector that
cannot be established at a depth now reports accept with `verified_depth` short of
it and the reason under `unresolved`, which separates cannot-establish from failed
and is the better shape. Reading the verdict alone scored 04 and 06 as separating
nothing. Measured properly the set is healthy: builder has two vectors, transitive
three, plus the accepting control.

docs/normative-crosswalk.md gains nine rows for spec 3.3.1, and its prose no longer
states requirements it does not cite. That guard is upstream's, contributed from
here as agentrust-io#175, and it was failing on this fork's own document.

603 pass; the three failures are the schema-const collision that predates this.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:UNKNOWN Contributor check flagged UNKNOWN risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants