test: rule registry the verifier consumes, and two independent vectors per rule - #148
Conversation
…s per rule Implements the #124 review as prescribed there. The reviewed design point: an obligation inventory recovered from the verifier's source by AST search stays blind to append/extend/constant/f-string refactors, so the safer shape is an explicit registry of named receipt rules that the verifier itself consumes, with mutation applied to named rule hooks rather than inferred patterns, and every registered rule held by at least two independent, load-bearing fixtures. This is that shape: - The verifier evaluates RULES, a registry of (code, severity, path, check) entries; _evaluate is the only emission point, and an AST guard fails on any code emitted around it. Statuses are a declared set enforced at construction. Behaviour on every existing fixture is unchanged. - The completeness suite mutates the registry by name: deleting a rule must change at least two fixtures' outcomes, and for every rule a declared weakened check - a plausible implementation shortcut - must separate its vectors, which is the issue's independence definition made executable. Both are fail-closed: a rule without two load-bearing vectors, or without a declared defect, fails the suite. A margins ratchet keeps coverage above the floor from silently thinning. - Fixtures 17-30 give every rule its second vector, each placed against a shortcut the first cannot detect: digests wrong only in the final character, case-variant identifiers, one-second freshness boundaries, a structurally well-formed signature of the wrong length, an explicit-null receipt, a self-report on the rejected branch. gen_rule_coverage_vectors.py regenerates 10-30 byte-for-byte; only public JWKs appear in the files. 240 tests pass; ruff and mypy clean. Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TGTKY8Fut5trnDKm8TReTa
|
Thank you for doing the rebase. That branch was 33 commits behind and I had left it without a PR on purpose, so this is work you took on that was mine. Confirming the half only the author can: I diffed the rebased head against One thing worth stating before it becomes a question. The branch carries 14 registered rules here and 22 on the fork's Your summary of the limit is accurate, including the part about a second independently written vector set. That one is still open. |
Upstream merged agentrust-io#148, the last large piece this fork was carrying, which the maintainer rebased and opened himself under this fork's authorship. With agentrust-io#122, agentrust-io#125, agentrust-io#126, agentrust-io#136 and agentrust-io#137 already merged, most of the thirty-three commits here described work that now lives upstream with better provenance than this fork can give it: a PR number and a maintainer's review. `git rebase upstream/main` was tried first and abandoned. It stopped on the oldest commit in the set, a schema-and-version alignment whose content upstream has since taken, superseded and released three times over. Replaying thirty-three commits against fifteen of upstream's resolves early commits into shapes that no longer mean anything. So: main reset to upstream/main, fork-only material re-applied. The old history is tagged `archive/pre-576507b` and pushed rather than discarded. What is held here, and why it is held: - agentrust-io#117 gap disclosure: design note, two normative drafts, 18 vectors, generator - agentrust-io#116 verifier compatibility: 8 vectors, generator, normative draft - the normative crosswalk, mapping every RFC 2119 statement to whom it binds - `docs/conformance-method.md`, `coverage-report/` (historical), DECISIONS.md - the independent signature path and the package-consistency test Each is an unaccepted proposal or a method write-up, not unfinished work. Four files needed a real merge, and not in the same direction. `models.py` and `__init__.py` are upstream's plus this fork's two profile constants, because upstream had moved on with `origin` (agentrust-io#135) and the `declared` enforcement mode (agentrust-io#143) and a wholesale copy would have dropped both. `sign.py` and `test_sign.py` are this fork's `accepted_profiles` version, which supersedes the minimal cutover check upstream took from agentrust-io#125 -- a supersession CLAUDE.md predicted when agentrust-io#125 was offered. All four merged cleanly three-way against a817621, the last commit the two histories agree on. The crosswalk guard earned its place in the same run: upstream added two normative statements this fork had never seen, and `test_normative_crosswalk.py` failed until both had rows. A source-derived inventory noticing its subject moved is the property that document exists to have. 432 passed, 1 skipped. ruff and mypy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
Upstream merged agentrust-io#148, the last large piece this fork was carrying, which the maintainer rebased and opened himself under this fork's authorship. With agentrust-io#122, described work that now lives upstream with better provenance than this fork can give it: a PR number and a maintainer's review. `git rebase upstream/main` was tried first and abandoned. It stopped on the oldest commit in the set, a schema-and-version alignment whose content upstream has since taken, superseded and released three times over. Replaying thirty-three commits against fifteen of upstream's resolves early commits into shapes that no longer mean anything. So: main reset to upstream/main, fork-only material re-applied. The old history is tagged `archive/pre-576507b` and pushed rather than discarded. What is held here, and why it is held: - agentrust-io#117 gap disclosure: design note, two normative drafts, 18 vectors, generator - agentrust-io#116 verifier compatibility: 8 vectors, generator, normative draft - the normative crosswalk, mapping every RFC 2119 statement to whom it binds - `docs/conformance-method.md`, `coverage-report/` (historical), DECISIONS.md - the independent signature path and the package-consistency test Each is an unaccepted proposal or a method write-up, not unfinished work. Four files needed a real merge, and not in the same direction. `models.py` and `__init__.py` are upstream's plus this fork's two profile constants, because upstream had moved on with `origin` (agentrust-io#135) and the `declared` enforcement mode (agentrust-io#143) and a wholesale copy would have dropped both. `sign.py` and `test_sign.py` are this fork's `accepted_profiles` version, which supersedes the minimal cutover check upstream took from agentrust-io#125 -- a supersession CLAUDE.md predicted when agentrust-io#125 was offered. All four merged cleanly three-way against a817621, the last commit the two histories agree on. The crosswalk guard earned its place in the same run: upstream added two normative statements this fork had never seen, and `test_normative_crosswalk.py` failed until both had rows. A source-derived inventory noticing its subject moved is the property that document exists to have. 432 passed, 1 skipped. ruff and mypy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
Closes #124. Authored by @lywinged, from their branch
lywinged/trace-spec:feat/124-registry-and-second-vectors. I rebased it onto current main (it was 33 commits behind) and droppeduv.lock; the commit is theirs and the authorship is intact. Opened as a maintainer because the branch had no PR.What it does. Every action-receipt obligation was exercised by exactly one fixture, so deleting, renaming or weakening any single vector silently retired a rule and nothing failed. This gives each of the 14 obligations a second, independent vector and makes the margin enforceable.
The design follows @carloshvp's review on the issue. The AST walk for string literals is retired as the inventory source, because
append,extend, constants and helper returns are silent blind spots. In its place the verifier consumes an explicit tuple of namedRuleentries and emits every code from a single loop, so an unregistered check is a check that never runs. The AST survives inverted: it now asserts that nothing else in the module appends to a failure or warning list, so an emission path bypassing the registry fails the suite.Independence is executable rather than asserted. The issue's definition is "one implementation defect makes one vector pass and the other fail", which only means something if the defects are named, so each rule declares at least one weakened variant of its check: digest-prefix comparison, case-normalised identifiers, clock tolerance, signature structure validated without cryptography. Two vectors count as independent only when a declared defect moves one and leaves the other alone, and a rule with no declared defect fails the suite.
tests/vector_margins.jsonis the ratchet: 14 rules, all at margin 2, so anything above the floor cannot decay back to it unnoticed.Verified on current main after the rebase: 313 passed, 1 skipped, ruff clean on
tests/andexamples/.One limit @lywinged states themselves, worth keeping visible: independence is relative to the declared defect set, so a shortcut nobody thought to declare is still invisible. It is a floor on discrimination, not a proof of it. They also noted they would rather the vectors were not all authored by one hand, and that a second independently written set would be more valuable as a cross-run than either set alone. That remains open and is not a reason to hold this.
🤖 Generated with Claude Code