Skip to content

test: rule registry the verifier consumes, and two independent vectors per rule - #148

Merged
imran-siddique merged 1 commit into
mainfrom
contrib/124-registry-and-second-vectors
Aug 10, 2026
Merged

test: rule registry the verifier consumes, and two independent vectors per rule#148
imran-siddique merged 1 commit into
mainfrom
contrib/124-registry-and-second-vectors

Conversation

@imran-siddique

Copy link
Copy Markdown
Member

Closes #124. Authored by @lywinged, from their branch lywinged/trace-spec:feat/124-registry-and-second-vectors. I rebased it onto current main (it was 33 commits behind) and dropped uv.lock; the commit is theirs and the authorship is intact. Opened as a maintainer because the branch had no PR.

What it does. Every action-receipt obligation was exercised by exactly one fixture, so deleting, renaming or weakening any single vector silently retired a rule and nothing failed. This gives each of the 14 obligations a second, independent vector and makes the margin enforceable.

The design follows @carloshvp's review on the issue. The AST walk for string literals is retired as the inventory source, because append, extend, constants and helper returns are silent blind spots. In its place the verifier consumes an explicit tuple of named Rule entries and emits every code from a single loop, so an unregistered check is a check that never runs. The AST survives inverted: it now asserts that nothing else in the module appends to a failure or warning list, so an emission path bypassing the registry fails the suite.

Independence is executable rather than asserted. The issue's definition is "one implementation defect makes one vector pass and the other fail", which only means something if the defects are named, so each rule declares at least one weakened variant of its check: digest-prefix comparison, case-normalised identifiers, clock tolerance, signature structure validated without cryptography. Two vectors count as independent only when a declared defect moves one and leaves the other alone, and a rule with no declared defect fails the suite.

tests/vector_margins.json is the ratchet: 14 rules, all at margin 2, so anything above the floor cannot decay back to it unnoticed.

Verified on current main after the rebase: 313 passed, 1 skipped, ruff clean on tests/ and examples/.

One limit @lywinged states themselves, worth keeping visible: independence is relative to the declared defect set, so a shortcut nobody thought to declare is still invisible. It is a floor on discrimination, not a proof of it. They also noted they would rather the vectors were not all authored by one hand, and that a second independently written set would be more valuable as a cross-run than either set alone. That remains open and is not a reason to hold this.

🤖 Generated with Claude Code

…s per rule

Implements the #124 review as prescribed there.

The reviewed design point: an obligation inventory recovered from the
verifier's source by AST search stays blind to append/extend/constant/f-string
refactors, so the safer shape is an explicit registry of named receipt rules
that the verifier itself consumes, with mutation applied to named rule hooks
rather than inferred patterns, and every registered rule held by at least two
independent, load-bearing fixtures. This is that shape:

- The verifier evaluates RULES, a registry of (code, severity, path, check)
  entries; _evaluate is the only emission point, and an AST guard fails on any
  code emitted around it. Statuses are a declared set enforced at construction.
  Behaviour on every existing fixture is unchanged.
- The completeness suite mutates the registry by name: deleting a rule must
  change at least two fixtures' outcomes, and for every rule a declared
  weakened check - a plausible implementation shortcut - must separate its
  vectors, which is the issue's independence definition made executable. Both
  are fail-closed: a rule without two load-bearing vectors, or without a
  declared defect, fails the suite. A margins ratchet keeps coverage above the
  floor from silently thinning.
- Fixtures 17-30 give every rule its second vector, each placed against a
  shortcut the first cannot detect: digests wrong only in the final character,
  case-variant identifiers, one-second freshness boundaries, a structurally
  well-formed signature of the wrong length, an explicit-null receipt, a
  self-report on the rejected branch. gen_rule_coverage_vectors.py regenerates
  10-30 byte-for-byte; only public JWKs appear in the files.

240 tests pass; ruff and mypy clean.

Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TGTKY8Fut5trnDKm8TReTa
@lywinged

Copy link
Copy Markdown
Collaborator

Thank you for doing the rebase. That branch was 33 commits behind and I had left it without a PR on purpose, so this is work you took on that was mine.

Confirming the half only the author can: I diffed the rebased head against feat/124-registry-and-second-vectors, and tests/test_action_receipt_fixtures.py, tests/test_vector_completeness.py and tests/vector_margins.json are byte-identical. You established that it runs; this is the other half, that it moved without changing.

One thing worth stating before it becomes a question. The branch carries 14 registered rules here and 22 on the fork's main. The extra 8 are the GapDisclosure rules from #117, an unaccepted proposal, so they are deliberately outside this PR and their fixtures sit under a separate proposal-117/ directory with their own numbering. The 14 here stand on their own.

Your summary of the limit is accurate, including the part about a second independently written vector set. That one is still open.

@imran-siddique
imran-siddique merged commit 576507b into main Aug 10, 2026
6 checks passed
@imran-siddique
imran-siddique deleted the contrib/124-registry-and-second-vectors branch August 10, 2026 04:11
lywinged added a commit to lywinged/trace-spec that referenced this pull request Aug 10, 2026
Upstream merged agentrust-io#148, the last large piece this fork was carrying, which the
maintainer rebased and opened himself under this fork's authorship. With agentrust-io#122,
agentrust-io#125, agentrust-io#126, agentrust-io#136 and agentrust-io#137 already merged, most of the thirty-three commits here
described work that now lives upstream with better provenance than this fork can
give it: a PR number and a maintainer's review.

`git rebase upstream/main` was tried first and abandoned. It stopped on the oldest
commit in the set, a schema-and-version alignment whose content upstream has since
taken, superseded and released three times over. Replaying thirty-three commits
against fifteen of upstream's resolves early commits into shapes that no longer
mean anything.

So: main reset to upstream/main, fork-only material re-applied. The old history is
tagged `archive/pre-576507b` and pushed rather than discarded.

What is held here, and why it is held:

- agentrust-io#117 gap disclosure: design note, two normative drafts, 18 vectors, generator
- agentrust-io#116 verifier compatibility: 8 vectors, generator, normative draft
- the normative crosswalk, mapping every RFC 2119 statement to whom it binds
- `docs/conformance-method.md`, `coverage-report/` (historical), DECISIONS.md
- the independent signature path and the package-consistency test

Each is an unaccepted proposal or a method write-up, not unfinished work.

Four files needed a real merge, and not in the same direction. `models.py` and
`__init__.py` are upstream's plus this fork's two profile constants, because
upstream had moved on with `origin` (agentrust-io#135) and the `declared` enforcement mode
(agentrust-io#143) and a wholesale copy would have dropped both. `sign.py` and `test_sign.py`
are this fork's `accepted_profiles` version, which supersedes the minimal cutover
check upstream took from agentrust-io#125 -- a supersession CLAUDE.md predicted when agentrust-io#125 was
offered. All four merged cleanly three-way against a817621, the last commit the
two histories agree on.

The crosswalk guard earned its place in the same run: upstream added two normative
statements this fork had never seen, and `test_normative_crosswalk.py` failed until
both had rows. A source-derived inventory noticing its subject moved is the
property that document exists to have.

432 passed, 1 skipped. ruff and mypy clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
lywinged added a commit to lywinged/trace-spec that referenced this pull request Aug 24, 2026
Upstream merged agentrust-io#148, the last large piece this fork was carrying, which the
maintainer rebased and opened himself under this fork's authorship. With agentrust-io#122,
described work that now lives upstream with better provenance than this fork can
give it: a PR number and a maintainer's review.

`git rebase upstream/main` was tried first and abandoned. It stopped on the oldest
commit in the set, a schema-and-version alignment whose content upstream has since
taken, superseded and released three times over. Replaying thirty-three commits
against fifteen of upstream's resolves early commits into shapes that no longer
mean anything.

So: main reset to upstream/main, fork-only material re-applied. The old history is
tagged `archive/pre-576507b` and pushed rather than discarded.

What is held here, and why it is held:

- agentrust-io#117 gap disclosure: design note, two normative drafts, 18 vectors, generator
- agentrust-io#116 verifier compatibility: 8 vectors, generator, normative draft
- the normative crosswalk, mapping every RFC 2119 statement to whom it binds
- `docs/conformance-method.md`, `coverage-report/` (historical), DECISIONS.md
- the independent signature path and the package-consistency test

Each is an unaccepted proposal or a method write-up, not unfinished work.

Four files needed a real merge, and not in the same direction. `models.py` and
`__init__.py` are upstream's plus this fork's two profile constants, because
upstream had moved on with `origin` (agentrust-io#135) and the `declared` enforcement mode
(agentrust-io#143) and a wholesale copy would have dropped both. `sign.py` and `test_sign.py`
are this fork's `accepted_profiles` version, which supersedes the minimal cutover
check upstream took from agentrust-io#125 -- a supersession CLAUDE.md predicted when agentrust-io#125 was
offered. All four merged cleanly three-way against a817621, the last commit the
two histories agree on.

The crosswalk guard earned its place in the same run: upstream added two normative
statements this fork had never seen, and `test_normative_crosswalk.py` failed until
both had rows. A source-derived inventory noticing its subject moved is the
property that document exists to have.

432 passed, 1 skipped. ruff and mypy clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(conformance): single-vector coverage has no margin — define and enforce independence

2 participants