test(conformance): adequacy criteria for a vector set, applied to every set here - #186
Merged
imran-siddique merged 1 commit intoAug 20, 2026
Merged
Conversation
…ry set here A conformance vector set is a claim that a non-implementing verifier will fail it. Nothing checked that claim. agentrust-io#169 and agentrust-io#170 were both found by asking it of a set rather than of a vector, and both were sets that were passing. Four criteria, each from a defect on a real set rather than from first principles: - a set must fail both unconditional implementations, accept-everything and reject-everything, or it pins nothing - each boundary needs more than one vector, since a single vector cannot distinguish a check that reads the head of a list from one that reads all of it - every set on disk is measured here or named with the test that measures it - shortfalls are recorded exactly, so they cannot widen unnoticed and the entry is deleted when someone closes the gap Applied to every set in this repository. `build-provenance-depth` carries a margin at every boundary. `canonicalization-boundary`, which I wrote, expects acceptance in every vector and so cannot tell a conformant verifier from one that accepts unconditionally; that is recorded rather than skipped, and the record is asserted so it cannot grow. `action-receipts` is named as covered by test_vector_completeness.py rather than graded twice. The completeness guard is on the instrument itself for a reason. SETS is a hand-maintained list of what gets graded, which is the defect these criteria exist to catch, and the one place it would otherwise be invisible: a set added later would simply not be graded and nothing would fail. Adding an unlisted set directory turns the guard red, as does a stale entry, as does naming a test that does not exist. Each was checked by making the change and watching the specific test fail. Signed-off-by: lywinged <louie.lunz@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
❔ Contributor Check: UNKNOWN
Automated check by AgenTrust Contributor Check. |
lywinged
added a commit
to lywinged/trace-spec
that referenced
this pull request
Aug 21, 2026
… merged agentrust-io#186 added criteria that every vector set on disk is measured against, and test_every_vector_set_on_disk_is_measured_somewhere fails for a set in neither SETS nor MEASURED_ELSEWHERE. This branch was opened three days before those criteria landed, so merging upstream leaves `delegation-link` as the one set nothing grades, and it is the only failure in the merged tree. Registered in SETS rather than named in MEASURED_ELSEWHERE, because the set holds up when it is actually graded rather than only pointed at: delegation-link: 23 vectors, 3 accepting, 10 distinct failure codes No shortfall on either criterion decidable from the fixtures. It is not satisfiable by an implementation that answers "accept" to everything or one that answers "reject" to everything, and every one of the ten failure codes is carried by exactly two vectors, which is the margin agentrust-io#124 asks for. Boundaries are counted by failure code, the default. adequacy.py says that assumption is the set's to justify: here the codes are the unit, because tests/delegation_margins.json records the per-code margin and tests/test_delegation_completeness.py holds each rule to being load-bearing for both of its vectors, deleting the rule from the registry rather than matching source text. The criteria adequacy.py leaves to each set, a rule nothing pins and a weakness shared across a boundary's vectors, are implemented there too, by rebuilding the registry without an entry and by substituting shortcut checks that read only the first link or the first hop. 605 passed, 1 skipped. Verified by removing the SETS entry again, which fails test_every_vector_set_on_disk_is_measured_somewhere on its own. Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
imran-siddique
pushed a commit
that referenced
this pull request
Aug 23, 2026
…with 23 conformance vectors (#184) * rfc: propose delegation-link verification, with the vectors that argue it The `delegation` block is normative in v0.2 and nothing says what a verifier does with a chain of them. `spec/trace-v0.2.md` never mentions `parent_record_hash` or `credential_id`; the only prose is one sentence in `docs/schema.md`, and every operative term in it is open — which bytes the digest covers, what "the delegation chain" is when no credential object exists in the schema, what a verifier does with a link it cannot compute. Two implementations can satisfy every constraint the repository states today and agree on nothing. `docs/rfcs/a2a-delegation-profile.md` proposes ten rules over the fields that already exist, so adopting it requires no schema change. `examples/delegation-link/` carries 23 vectors that score an implementation against them. Requirement keywords in the RFC are lowercase on purpose: a proposal that writes itself in the imperative is a specification nobody agreed to. Three forks in the current text had to be settled before a single vector could be written, and each is recorded with its reason rather than assumed: The digest covers the complete parent record, signature included. A digest over the signed body alone does not bind the parent's *signer* — anyone may re-sign identical bytes under another key and satisfy the child's commitment — so the child would have committed to what its parent said and not to who said it. Vector 05 is a complete, correctly signed chain whose only defect is which bytes its link was computed over. There is no cycle rule. A cycle needs each record's block to carry a digest covering the block that names it back, which is a hash collision; a rule against it would be untestable by construction. The reachable analogue is an unbounded chain, and that is the only reason the depth bound exists. Stated so a reader can tell which of the two was decided and which was forgotten. A link naming a digest algorithm the verifier cannot compute makes the chain unverifiable, not invalid. Reporting `parent_not_found` for it would be a finding nobody made: the verifier did not fail to find the parent, it did not look. This is the delegation-surface instance of the semantics merged in `docs/verification.md`, and `parent_not_found` is explicitly guarded on algorithm support so the two cannot be produced together for one link. Coverage is held to #124's discipline from the first vector rather than as a later hardening pass: two load-bearing vectors per rule, and for every rule at least one declared implementation defect that one vector catches and the other misses. All ten defects model a real shortcut — verifying the leaf only, anchoring on any trusted key found, an off-by-one bound, case-insensitive lookup of an opaque identifier, issuer and holder compared to the wrong ends of the hop, half a validity window, narrowing checked at one hop, the link algorithm read once and assumed uniform. Two of those declarations found faults in the walk while it was being written, which is the argument for declaring them rather than asserting margin and stopping. The walk's break condition originally repeated the depth comparison, so a weakened bound never got to walk further than a correct one and both depth vectors moved together under every mutation — margin without independence. And an earlier vector 09 put an untrusted root three hops down, which no defect could separate from vector 08; the version that separates them places a *trusted* key partway up the chain, which is the shortcut an implementation actually takes. Reproducibility is a property of the corpus, not a courtesy. Keys derive from one published seed by role label. `tests/test_generators_reproduce_fixtures.py` (#171) discovered the generator with no new guard code and holds it to byte reproduction with no entry in the `NOT_GENERATED` ledger, which is the bar #178 proposes for the repository's corpora. Every record in every vector, including the ones built to fail, validates against `schema/trace-claim.json`: a defect the schema already rejects is not a profile defect, and a rule that looks covered only because its vector is malformed in some louder way is not covered. Nothing enters the package's public API. The walk lives in `tests/`, beside the action-receipt verifier it is modelled on, because the rules it implements are not normative yet. Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com> * rfc: record what running the corpus through cA2A returned The proposal argued for cross-verification and did not do any, which left its central section a plan. This runs the 23 vectors against `ca2a_verify.verify_trace_dag` at ca2a 5dd77b2 and writes down what came back, including the parts that went against the draft. The §4.1 digest decision is confirmed from outside this repository. `ca2a_runtime.trace_binding.trace_record_hash` computes the sha256 of the complete signed record's RFC 8785 bytes — byte-identical to what the profile specifies, arrived at separately. Vectors 01-07 agree in verdict and in reason. Vectors 22 and 23 disagree exactly as §4.3 predicted: cA2A accepts a `sha384:` link at block validation, compares it against a hash it only ever computes as `sha256:`, and reports the chain as "a tampered or reparented record". An intact chain addressed under the other permitted algorithm is reported as tampering. The distinction between unreadable and contradicted is now observed rather than argued. Two things the draft got wrong, corrected here rather than left standing: It said cA2A "states that its credentials are cross-verifiable with agent-manifest" and that nothing tests the claim. The claim in `ca2a_runtime/canonical.py` is narrower — that RFC 8785 makes the signed byte string identical across conforming implementations, so signatures verify either side. Read as credential interoperability it is a claim ca2a does not make. Checked on the axis it does make: ca2a hand-implements JCS rather than taking a library, and that implementation is byte-identical to the reference on all four vectors of `examples/canonicalization-boundary/`, both UTF-16 key-order cases included. Upheld. A first pass recorded that cA2A has no depth bound. It has one — `max_depth`, default 8, on the credential chain rather than on the record DAG. A bound in a different place is not an absent bound. The credential surfaces turn out not to be comparable at all, which is the finding rather than an obstacle to it: three repositories, three delegation models, no conversion between them. §7.1 tabulates them. Two consequences worth carrying forward — agent-manifest already narrows on `data_classifications`, which is independent support for D-9 belonging on this surface, and cA2A credentials carry no validity window at all, so D-8 has no counterpart there. The trust contract also differs and cannot be normalised away: `verify_trace_dag` requires every record's key to be trusted, this profile anchors on the root's. Under this profile's contract cA2A rejects every valid chain longer than one record. Neither is wrong; they fit different deployments, and cA2A itself uses the root-anchored model on its other surface. No code changes. The vectors are untouched and both suites still pass. Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com> * rfc: run the corpus in the reverse direction, and read agent-manifest's outcomes The cross-check so far only pushed this corpus outward, which shows that cA2A rejects what the profile rejects and nothing about whether the profile describes what the ecosystem emits. This runs it the other way and reads the third implementation's declared outcomes. `ca2a/examples/trace-dag/demo.py` emits a signed three-hop TRACE DAG through cA2A's own `trace_binding`. Against it: three schema-valid records, both links matching the section 4.1 preimage exactly, all three signatures valid under D-1, and the chain returns `verified` with no codes and no adjustment to the walk. A chain produced by an independent implementation verifies here unchanged. agent-manifest turns out to settle section 4.3 rather than leave it open. Its corpus declares results as data in the vector files -- VALID, MISMATCH, UNVERIFIABLE, EXPIRED, REVOKED, SIGNATURE_MISSING, INCOMPLETE, INCOMPATIBLE_VERSION, ATTESTATION_UNAVAILABLE -- and AM-VEC-012 declares `{"result": "UNVERIFIABLE", "fields_verified": {"delegation_chain": "UNVERIFIABLE"}}` for a delegation chain with no public keys. Evidence the verifier lacks what it needs to check, recorded as unreadable rather than as a finding against the chain: section 4.3, on this surface, in a second implementation, arrived at independently. Two of the three distinguish unreadable from contradicted; cA2A's TRACE DAG verifier collapses them, which makes the sha384 divergence a gap rather than a preference. Its `fields_verified` shape is prior art this proposal does not have. A verdict per field says more than a verdict per chain, and section 8 should probably ask about it. Two smaller things recorded where they were found. `examples/trace-dag/` commits a README and a demo but no vectors -- the DAG is produced at runtime and not kept, which is the gap this corpus fills from the trace-spec side. And cA2A uses the field name `parent_record_hash` in two formats: the schema's prefixed digest in a TRACE record, and a bare hex digest in its own provenance DAG, on records carrying no TRACE fields. Both deliberate, neither wrong, and a hazard for anyone writing a parser against the name. No code changes; the vectors are untouched and both suites pass. Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com> * rfc: name the two things a Project Lead would notice first Both are gaps in this document rather than in anything it argues, and both were found by reading `ROADMAP.md:21` against §6 rather than by anyone raising them. **The mutual case.** That line scopes the v0.3 A2A profile as "binding rules over the `delegation` block ... including the mutual case". §6 lists six things this proposal does not do and omitted the one the roadmap names. Nothing here covers mutual delegation: every rule walks one chain in one direction, and the block as it stands names one parent and no peer. Calling two agents each holding the other's authority "two chains" would be deciding that question rather than raising it, so §6 now says so plainly. It is the largest distance between the roadmap's line and this document. **Who this is for.** The same line names cA2A as the reference implementation and says nothing about who writes the binding rules, and this was written without asking. §8 opens with that question ahead of the design ones, because the answer changes what the document should become: the profile itself would need the mutual case, a credential model and a ratification path; an input stays a set of rules with executable material behind them, liftable or discardable a rule at a time. Neither is a change to a rule, a vector or a suite. 585 passed. * rfc: state the method, which was the reason for the order and went unwritten The document presented the three decisions in §4 as decisions and never said how they were arrived at. They were not read out of `docs/schema.md`; they were hit, because no vector could be written without settling them, and in each case the text supports both branches. A reader passes over all three without noticing. Someone building a fixture cannot get to the end of one. That order -- corpus first, and let it interrogate the text -- is the part worth keeping if every rule here is replaced, because it yields a measurement this repository does not otherwise have. Not whether tests cover the rules, which measures an implementation, but whether two independent readings of the same normative text produce the same rules, which measures the specification. Agreement means the text is doing its job; divergence names the sentence that is missing. §7.1 was already that measurement run once and was not labelled as one. Two implementations written without reference to this document agree with §4.1 and with each other on the digest preimage, and split on the unresolvable-algorithm question -- one calling it unreadable, one calling it tampering. The first result is the text working. The second is a located gap that took no argument to find, because the same question was put to two implementations rather than debated. It also settles what a second profile design would be for. One reading measures nothing, so an independently written set of binding rules is the experiment, not a collision with this one. No rule, vector or suite changed. 585 passed. * rfc: correct what the roadmap's "mutual case" refers to The previous entry read that line as mutual delegation and reported this document as short of it. That was a guess at the referent, made without checking, and it is wrong. In the reference implementation the mutual case is mutual attestation. ca2a/docs/spec/mutual-attestation.md describes a callee-issued challenge and a caller offer bound to it, so each side establishes what the other is running before a payload opens. It separates the two concerns explicitly -- it "establishes what each side is running", while "the delegation chain remains the thing that says what it is allowed to ask for" -- and it does not mention a Trust Record anywhere. Nor does cA2A's own docs/spec/trace-a2a-profile.md, whose A2A profile is the delegation-link block and nothing else. Which changes what the entry says. The roadmap asks the A2A profile to cover something with no record representation today in either repository, sitting at the transport layer rather than on this surface. That is a scoping question -- either mutual attestation gains a binding into the record, which is a schema question rather than a verification one, or the v0.3 profile is two profiles -- and not a coverage failure in these rules. A reader comparing this document against that roadmap line would otherwise conclude the second. The bidirectional-delegation reading is kept as a separate note rather than dropped, because it is true and unreachable for the reason section 4.2 gives, and because the two readings should not merge later. Every claim above traced to its file before writing: grep for "trace" in mutual-attestation.md returns 0, grep for "mutual" in trace-a2a-profile.md returns 0, and both quotations were checked against the source with whitespace normalised, since the file wraps mid-sentence and a single-line grep misses them. No rule, vector or suite changed. 585 passed, ruff clean. * test(adequacy): grade delegation-link by the criteria this repository merged #186 added criteria that every vector set on disk is measured against, and test_every_vector_set_on_disk_is_measured_somewhere fails for a set in neither SETS nor MEASURED_ELSEWHERE. This branch was opened three days before those criteria landed, so merging upstream leaves `delegation-link` as the one set nothing grades, and it is the only failure in the merged tree. Registered in SETS rather than named in MEASURED_ELSEWHERE, because the set holds up when it is actually graded rather than only pointed at: delegation-link: 23 vectors, 3 accepting, 10 distinct failure codes No shortfall on either criterion decidable from the fixtures. It is not satisfiable by an implementation that answers "accept" to everything or one that answers "reject" to everything, and every one of the ten failure codes is carried by exactly two vectors, which is the margin #124 asks for. Boundaries are counted by failure code, the default. adequacy.py says that assumption is the set's to justify: here the codes are the unit, because tests/delegation_margins.json records the per-code margin and tests/test_delegation_completeness.py holds each rule to being load-bearing for both of its vectors, deleting the rule from the registry rather than matching source text. The criteria adequacy.py leaves to each set, a rule nothing pins and a weakness shared across a boundary's vectors, are implemented there too, by rebuilding the registry without an entry and by substituting shortcut checks that read only the first link or the first hop. 605 passed, 1 skipped. Verified by removing the SETS entry again, which fails test_every_vector_set_on_disk_is_measured_somewhere on its own. Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com> * rfc: the sha384 divergence is closed, so stop reporting it as live Section 7.1 recorded the cA2A disagreement in the present tense: its block validator accepts a sha384: link, compares it against a hash it only computes as sha256:, and reports ProvenanceLinkBroken with "a tampered or reparented record was detected". That was true when the corpus was run and is not true now, so the document was carrying a defect report against another repository that the other repository has already fixed. Checked at ca2a 52141e8 rather than taken from the report: src/ca2a_verify/dag.py:194 raises TraceDigestUnsupported with the detail "the chain is unverifiable here, not invalid" at line 199, and the parent-link comparison that produced ProvenanceLinkBroken is at line 201, after it. So the guard precedes the comparison and vectors 22 and 23 now describe fixed behaviour. Both places are re-tensed rather than deleted. What the case establishes is not that one verifier had a bug: a corpus written to argue a rule found the case, the other implementation changed, and the shape it changed to is the distinction section 4.3 asks for. Deleting it would drop the strongest evidence in the document that the corpus does what it claims. The second passage said two of three implementations distinguish unreadable from contradicted; it is now all three. Section 4.3's rule text is untouched, since it states the rule rather than reporting on an implementation. 605 passed, 1 skipped. Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com> --------- Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
lywinged
added a commit
to lywinged/trace-spec
that referenced
this pull request
Aug 24, 2026
…ded references Rebased onto upstream/main, 26 commits behind. Six files were flagged by the overlap check and three needed a real merge; the directions are the ones DECISIONS.md recorded on 2026-08-10, because the situation is the same one. models.py and __init__.py are upstream's plus this fork's two profile constants and the accepted_profiles machinery. sign.py merged cleanly as this fork's richer version, superseding the minimal cutover check upstream took from agentrust-io#125, which CLAUDE.md section 5 predicted when agentrust-io#125 was offered. The adequacy criteria are now upstream's, from agentrust-io#186, and the fork's own copy is dropped rather than merged: upstream's went through review, carries the completeness guard the fork's did not, and gained a delegation-link set from agentrust-io#184. Only the verifier-compatibility loader is re-applied, since that set is fork-only, along with its recorded shortfall. The pyproject pythonpath entry goes with it: it existed for a bare `from adequacy import` that upstream's version does not use. Upstream's models.py docstring says "TRACE v0.1 Trust Record" on a class whose eat_profile is Literal["...trace-v0.2"]. The fork's correction is kept. The crosswalk guard did what it exists to do. Upstream replaced "Verifiers MUST consult current revocation status" with a SCITT-anchored model and added the references block, and the source-derived inventory failed until nine rows moved. The four references rows are mostly absences: an assurance-neutral block staying neutral is discharged by nothing deriving a platform value from it, and two of the four MUST NOTs bind verifier behaviour that no record shape can express, which the schema description already says. The revocation rows are the ones worth reading. This library implements the fallback in 3.2.3 and not the primary rule: there is no last_valid_entry_id, no log_id and no receipt comparison in src/, so a record is rejected on a key match whether or not it carries an anchor. Section 3.2.3 names that behaviour itself for deployments with no receipts, so it is incomplete rather than wrong, and the row says which. 856 passed, 1 skipped; ruff and mypy clean. Each new row and the restored shortfall entry were checked by mutation: deleting a row, misspelling an anchor, and dropping a recorded boundary each turn the naming test red. Signed-off-by: lywinged <louie.lunz@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A conformance vector set is a claim: a verifier that does not implement these rules
will fail this set. Nothing in the repository checked that claim, and it is not implied
by the vectors passing.
#169 and #170 both came out of asking it. Both were sets that were green.
build-provenance-depthreported the same verdict at every depth, so averifier that resolved one link and assumed the rest matched passed it.
canonicalization-boundaryseparated its closest form by a single vector,and that vector put the divergence at the top level, where the obvious implementation
already sorts.
Neither was found by reading a vector. Both were found by asking what a set fails to
distinguish.
The criteria
Four, each from a defect on a real set rather than from first principles:
What it says about this repository today
Applied to every set, by the same loader, with the results recorded where they fall.
canonicalization-boundaryis mine. Every vector in it expects acceptance, so theset cannot tell a conformant verifier from one that accepts unconditionally. That second
implementation is a real failure mode, not a hypothetical, so it is a gap and not a
design. It is recorded in
KNOWN_ONE_DIRECTIONALwith the record asserted, so it cannotwiden quietly, and the entry is deleted when the set gains a record signed over a non-JCS
form that a conformant verifier must reject.
I mention this first because a standard that only ever measures other people's work is
advocacy. The only shortfall this currently records is on a set I wrote.
The guard on the instrument itself
SETSis a hand-maintained list of what gets graded, which is the third criterion'sdefect, sitting in the one place it would otherwise be invisible: a set added later would
simply not be graded, and nothing would fail to say so.
test_every_vector_set_on_disk_is_measured_somewherecomparesSETSandMEASURED_ELSEWHEREagainst what is actually inexamples/, in both directions, since astale entry hides a renamed set as effectively as a missing one.
Rather than assert these fail for the right reason, I broke each and recorded which test
went red:
test_every_vector_set_on_disk_is_measured_somewhereMEASURED_ELSEWHEREnames a test file that does not existtest_the_test_named_as_measuring_a_set_elsewhere_existsWhat this costs, said before it happens rather than after
A future PR that adds a thin vector set will go red here, and the failure will name
the boundary and the vector rather than a line number. That is the intended behaviour and
it is a policy consequence of merging a test, so it should be a decision rather than a
surprise. The escape hatch is deliberate and visible: record the shortfall in
KNOWN_THINwith its exact extent, which documents the gap instead of hiding it, and theentry is deleted when someone writes the second vector.
Scope
Tests and an informative document. No normative text, no RFC 2119 keyword, no schema
change, so no sponsor is needed under
GOVERNANCE.md. Same class as #169, #171 and #175.485 passed, 1 skipped;ruff check src tests scriptsclean;mypyclean. Branched offmainat697e20a.The criteria are also usable outside this repository, since they read
expectedandnothing else. If they are more useful in trace-tests than here, say so and I will move
them.