Skip to content

test(conformance): adequacy criteria for a vector set, applied to every set here - #186

Merged
imran-siddique merged 1 commit into
agentrust-io:mainfrom
lywinged:test/vector-set-adequacy
Aug 20, 2026
Merged

test(conformance): adequacy criteria for a vector set, applied to every set here#186
imran-siddique merged 1 commit into
agentrust-io:mainfrom
lywinged:test/vector-set-adequacy

Conversation

@lywinged

Copy link
Copy Markdown
Collaborator

A conformance vector set is a claim: a verifier that does not implement these rules
will fail this set.
Nothing in the repository checked that claim, and it is not implied
by the vectors passing.

#169 and #170 both came out of asking it. Both were sets that were green.

Neither was found by reading a vector. Both were found by asking what a set fails to
distinguish.

The criteria

Four, each from a defect on a real set rather than from first principles:

Criterion The defect behind it
A set must fail both unconditional implementations a set of all-rejections is passed by a verifier that rejects everything
Every boundary needs more than one vector one vector cannot separate a check that reads the head of a list from one that reads all of it
Every set on disk is measured, or named with the test that measures it a hand-maintained list of what gets graded silently stops being complete
Shortfalls are recorded exactly a gap that is skipped can widen; a gap asserted to its exact extent cannot

What it says about this repository today

Applied to every set, by the same loader, with the results recorded where they fall.

build-provenance-depth      margin at every boundary, nothing recorded
canonicalization-boundary   one-directional: every vector expects acceptance
action-receipts             covered by tests/test_vector_completeness.py

canonicalization-boundary is mine. Every vector in it expects acceptance, so the
set cannot tell a conformant verifier from one that accepts unconditionally. That second
implementation is a real failure mode, not a hypothetical, so it is a gap and not a
design. It is recorded in KNOWN_ONE_DIRECTIONAL with the record asserted, so it cannot
widen quietly, and the entry is deleted when the set gains a record signed over a non-JCS
form that a conformant verifier must reject.

I mention this first because a standard that only ever measures other people's work is
advocacy. The only shortfall this currently records is on a set I wrote.

The guard on the instrument itself

SETS is a hand-maintained list of what gets graded, which is the third criterion's
defect, sitting in the one place it would otherwise be invisible: a set added later would
simply not be graded, and nothing would fail to say so.

test_every_vector_set_on_disk_is_measured_somewhere compares SETS and
MEASURED_ELSEWHERE against what is actually in examples/, in both directions, since a
stale entry hides a renamed set as effectively as a missing one.

Rather than assert these fail for the right reason, I broke each and recorded which test
went red:

Mutation Test that caught it
an unlisted set directory appears test_every_vector_set_on_disk_is_measured_somewhere
an entry names a set that is not on disk the same
the directory scan stops finding anything the same
MEASURED_ELSEWHERE names a test file that does not exist test_the_test_named_as_measuring_a_set_elsewhere_exists

What this costs, said before it happens rather than after

A future PR that adds a thin vector set will go red here, and the failure will name
the boundary and the vector rather than a line number. That is the intended behaviour and
it is a policy consequence of merging a test, so it should be a decision rather than a
surprise. The escape hatch is deliberate and visible: record the shortfall in
KNOWN_THIN with its exact extent, which documents the gap instead of hiding it, and the
entry is deleted when someone writes the second vector.

Scope

Tests and an informative document. No normative text, no RFC 2119 keyword, no schema
change, so no sponsor is needed under GOVERNANCE.md. Same class as #169, #171 and #175.

485 passed, 1 skipped; ruff check src tests scripts clean; mypy clean. Branched off
main at 697e20a.

The criteria are also usable outside this repository, since they read expected and
nothing else. If they are more useful in trace-tests than here, say so and I will move
them.

…ry set here

A conformance vector set is a claim that a non-implementing verifier will fail it.
Nothing checked that claim. agentrust-io#169 and agentrust-io#170 were both found by asking it of a set
rather than of a vector, and both were sets that were passing.

Four criteria, each from a defect on a real set rather than from first principles:

- a set must fail both unconditional implementations, accept-everything and
  reject-everything, or it pins nothing
- each boundary needs more than one vector, since a single vector cannot
  distinguish a check that reads the head of a list from one that reads all of it
- every set on disk is measured here or named with the test that measures it
- shortfalls are recorded exactly, so they cannot widen unnoticed and the entry is
  deleted when someone closes the gap

Applied to every set in this repository. `build-provenance-depth` carries a margin
at every boundary. `canonicalization-boundary`, which I wrote, expects acceptance in
every vector and so cannot tell a conformant verifier from one that accepts
unconditionally; that is recorded rather than skipped, and the record is asserted so
it cannot grow. `action-receipts` is named as covered by test_vector_completeness.py
rather than graded twice.

The completeness guard is on the instrument itself for a reason. SETS is a
hand-maintained list of what gets graded, which is the defect these criteria exist
to catch, and the one place it would otherwise be invisible: a set added later would
simply not be graded and nothing would fail. Adding an unlisted set directory turns
the guard red, as does a stale entry, as does naming a test that does not exist.
Each was checked by making the change and watching the specific test fail.

Signed-off-by: lywinged <louie.lunz@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Contributor Check: UNKNOWN

Check Result
Profile UNKNOWN
Credential LOW
Overall UNKNOWN

Automated check by AgenTrust Contributor Check.

@github-actions github-actions Bot added the needs-review:UNKNOWN Contributor check flagged UNKNOWN risk label Aug 18, 2026
@imran-siddique
imran-siddique merged commit 1f76106 into agentrust-io:main Aug 20, 2026
6 checks passed
lywinged added a commit to lywinged/trace-spec that referenced this pull request Aug 21, 2026
… merged

agentrust-io#186 added criteria that every vector set on disk is measured against, and
test_every_vector_set_on_disk_is_measured_somewhere fails for a set in neither
SETS nor MEASURED_ELSEWHERE. This branch was opened three days before those
criteria landed, so merging upstream leaves `delegation-link` as the one set
nothing grades, and it is the only failure in the merged tree.

Registered in SETS rather than named in MEASURED_ELSEWHERE, because the set
holds up when it is actually graded rather than only pointed at:

    delegation-link: 23 vectors, 3 accepting, 10 distinct failure codes

No shortfall on either criterion decidable from the fixtures. It is not
satisfiable by an implementation that answers "accept" to everything or one
that answers "reject" to everything, and every one of the ten failure codes is
carried by exactly two vectors, which is the margin agentrust-io#124 asks for.

Boundaries are counted by failure code, the default. adequacy.py says that
assumption is the set's to justify: here the codes are the unit, because
tests/delegation_margins.json records the per-code margin and
tests/test_delegation_completeness.py holds each rule to being load-bearing for
both of its vectors, deleting the rule from the registry rather than matching
source text. The criteria adequacy.py leaves to each set, a rule nothing pins
and a weakness shared across a boundary's vectors, are implemented there too,
by rebuilding the registry without an entry and by substituting shortcut checks
that read only the first link or the first hop.

605 passed, 1 skipped. Verified by removing the SETS entry again, which fails
test_every_vector_set_on_disk_is_measured_somewhere on its own.

Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
imran-siddique pushed a commit that referenced this pull request Aug 23, 2026
…with 23 conformance vectors (#184)

* rfc: propose delegation-link verification, with the vectors that argue it

The `delegation` block is normative in v0.2 and nothing says what a verifier does
with a chain of them. `spec/trace-v0.2.md` never mentions `parent_record_hash` or
`credential_id`; the only prose is one sentence in `docs/schema.md`, and every
operative term in it is open — which bytes the digest covers, what "the delegation
chain" is when no credential object exists in the schema, what a verifier does with
a link it cannot compute. Two implementations can satisfy every constraint the
repository states today and agree on nothing.

`docs/rfcs/a2a-delegation-profile.md` proposes ten rules over the fields that
already exist, so adopting it requires no schema change. `examples/delegation-link/`
carries 23 vectors that score an implementation against them. Requirement keywords
in the RFC are lowercase on purpose: a proposal that writes itself in the imperative
is a specification nobody agreed to.

Three forks in the current text had to be settled before a single vector could be
written, and each is recorded with its reason rather than assumed:

  The digest covers the complete parent record, signature included. A digest over
  the signed body alone does not bind the parent's *signer* — anyone may re-sign
  identical bytes under another key and satisfy the child's commitment — so the
  child would have committed to what its parent said and not to who said it.
  Vector 05 is a complete, correctly signed chain whose only defect is which bytes
  its link was computed over.

  There is no cycle rule. A cycle needs each record's block to carry a digest
  covering the block that names it back, which is a hash collision; a rule against
  it would be untestable by construction. The reachable analogue is an unbounded
  chain, and that is the only reason the depth bound exists. Stated so a reader can
  tell which of the two was decided and which was forgotten.

  A link naming a digest algorithm the verifier cannot compute makes the chain
  unverifiable, not invalid. Reporting `parent_not_found` for it would be a finding
  nobody made: the verifier did not fail to find the parent, it did not look. This
  is the delegation-surface instance of the semantics merged in
  `docs/verification.md`, and `parent_not_found` is explicitly guarded on algorithm
  support so the two cannot be produced together for one link.

Coverage is held to #124's discipline from the first vector rather than as a later
hardening pass: two load-bearing vectors per rule, and for every rule at least one
declared implementation defect that one vector catches and the other misses. All ten
defects model a real shortcut — verifying the leaf only, anchoring on any trusted key
found, an off-by-one bound, case-insensitive lookup of an opaque identifier, issuer
and holder compared to the wrong ends of the hop, half a validity window, narrowing
checked at one hop, the link algorithm read once and assumed uniform.

Two of those declarations found faults in the walk while it was being written, which
is the argument for declaring them rather than asserting margin and stopping. The
walk's break condition originally repeated the depth comparison, so a weakened bound
never got to walk further than a correct one and both depth vectors moved together
under every mutation — margin without independence. And an earlier vector 09 put an
untrusted root three hops down, which no defect could separate from vector 08; the
version that separates them places a *trusted* key partway up the chain, which is
the shortcut an implementation actually takes.

Reproducibility is a property of the corpus, not a courtesy. Keys derive from one
published seed by role label. `tests/test_generators_reproduce_fixtures.py` (#171)
discovered the generator with no new guard code and holds it to byte reproduction
with no entry in the `NOT_GENERATED` ledger, which is the bar #178 proposes for the
repository's corpora.

Every record in every vector, including the ones built to fail, validates against
`schema/trace-claim.json`: a defect the schema already rejects is not a profile
defect, and a rule that looks covered only because its vector is malformed in some
louder way is not covered.

Nothing enters the package's public API. The walk lives in `tests/`, beside the
action-receipt verifier it is modelled on, because the rules it implements are not
normative yet.

Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>

* rfc: record what running the corpus through cA2A returned

The proposal argued for cross-verification and did not do any, which left its
central section a plan. This runs the 23 vectors against
`ca2a_verify.verify_trace_dag` at ca2a 5dd77b2 and writes down what came back,
including the parts that went against the draft.

The §4.1 digest decision is confirmed from outside this repository.
`ca2a_runtime.trace_binding.trace_record_hash` computes the sha256 of the
complete signed record's RFC 8785 bytes — byte-identical to what the profile
specifies, arrived at separately. Vectors 01-07 agree in verdict and in reason.

Vectors 22 and 23 disagree exactly as §4.3 predicted: cA2A accepts a `sha384:`
link at block validation, compares it against a hash it only ever computes as
`sha256:`, and reports the chain as "a tampered or reparented record". An intact
chain addressed under the other permitted algorithm is reported as tampering.
The distinction between unreadable and contradicted is now observed rather than
argued.

Two things the draft got wrong, corrected here rather than left standing:

  It said cA2A "states that its credentials are cross-verifiable with
  agent-manifest" and that nothing tests the claim. The claim in
  `ca2a_runtime/canonical.py` is narrower — that RFC 8785 makes the signed byte
  string identical across conforming implementations, so signatures verify
  either side. Read as credential interoperability it is a claim ca2a does not
  make. Checked on the axis it does make: ca2a hand-implements JCS rather than
  taking a library, and that implementation is byte-identical to the reference
  on all four vectors of `examples/canonicalization-boundary/`, both UTF-16
  key-order cases included. Upheld.

  A first pass recorded that cA2A has no depth bound. It has one — `max_depth`,
  default 8, on the credential chain rather than on the record DAG. A bound in
  a different place is not an absent bound.

The credential surfaces turn out not to be comparable at all, which is the
finding rather than an obstacle to it: three repositories, three delegation
models, no conversion between them. §7.1 tabulates them. Two consequences worth
carrying forward — agent-manifest already narrows on `data_classifications`,
which is independent support for D-9 belonging on this surface, and cA2A
credentials carry no validity window at all, so D-8 has no counterpart there.

The trust contract also differs and cannot be normalised away: `verify_trace_dag`
requires every record's key to be trusted, this profile anchors on the root's.
Under this profile's contract cA2A rejects every valid chain longer than one
record. Neither is wrong; they fit different deployments, and cA2A itself uses
the root-anchored model on its other surface.

No code changes. The vectors are untouched and both suites still pass.

Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>

* rfc: run the corpus in the reverse direction, and read agent-manifest's outcomes

The cross-check so far only pushed this corpus outward, which shows that cA2A
rejects what the profile rejects and nothing about whether the profile describes
what the ecosystem emits. This runs it the other way and reads the third
implementation's declared outcomes.

`ca2a/examples/trace-dag/demo.py` emits a signed three-hop TRACE DAG through
cA2A's own `trace_binding`. Against it: three schema-valid records, both links
matching the section 4.1 preimage exactly, all three signatures valid under D-1,
and the chain returns `verified` with no codes and no adjustment to the walk. A
chain produced by an independent implementation verifies here unchanged.

agent-manifest turns out to settle section 4.3 rather than leave it open. Its
corpus declares results as data in the vector files -- VALID, MISMATCH,
UNVERIFIABLE, EXPIRED, REVOKED, SIGNATURE_MISSING, INCOMPLETE,
INCOMPATIBLE_VERSION, ATTESTATION_UNAVAILABLE -- and AM-VEC-012 declares
`{"result": "UNVERIFIABLE", "fields_verified": {"delegation_chain":
"UNVERIFIABLE"}}` for a delegation chain with no public keys. Evidence the
verifier lacks what it needs to check, recorded as unreadable rather than as a
finding against the chain: section 4.3, on this surface, in a second
implementation, arrived at independently. Two of the three distinguish
unreadable from contradicted; cA2A's TRACE DAG verifier collapses them, which
makes the sha384 divergence a gap rather than a preference.

Its `fields_verified` shape is prior art this proposal does not have. A verdict
per field says more than a verdict per chain, and section 8 should probably ask
about it.

Two smaller things recorded where they were found. `examples/trace-dag/` commits
a README and a demo but no vectors -- the DAG is produced at runtime and not
kept, which is the gap this corpus fills from the trace-spec side. And cA2A uses
the field name `parent_record_hash` in two formats: the schema's prefixed digest
in a TRACE record, and a bare hex digest in its own provenance DAG, on records
carrying no TRACE fields. Both deliberate, neither wrong, and a hazard for
anyone writing a parser against the name.

No code changes; the vectors are untouched and both suites pass.

Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>

* rfc: name the two things a Project Lead would notice first

Both are gaps in this document rather than in anything it argues, and both were
found by reading `ROADMAP.md:21` against §6 rather than by anyone raising them.

**The mutual case.** That line scopes the v0.3 A2A profile as "binding rules over
the `delegation` block ... including the mutual case". §6 lists six things this
proposal does not do and omitted the one the roadmap names. Nothing here covers
mutual delegation: every rule walks one chain in one direction, and the block as
it stands names one parent and no peer. Calling two agents each holding the
other's authority "two chains" would be deciding that question rather than
raising it, so §6 now says so plainly. It is the largest distance between the
roadmap's line and this document.

**Who this is for.** The same line names cA2A as the reference implementation and
says nothing about who writes the binding rules, and this was written without
asking. §8 opens with that question ahead of the design ones, because the answer
changes what the document should become: the profile itself would need the mutual
case, a credential model and a ratification path; an input stays a set of rules
with executable material behind them, liftable or discardable a rule at a time.

Neither is a change to a rule, a vector or a suite. 585 passed.

* rfc: state the method, which was the reason for the order and went unwritten

The document presented the three decisions in §4 as decisions and never said how
they were arrived at. They were not read out of `docs/schema.md`; they were hit,
because no vector could be written without settling them, and in each case the
text supports both branches. A reader passes over all three without noticing.
Someone building a fixture cannot get to the end of one.

That order -- corpus first, and let it interrogate the text -- is the part worth
keeping if every rule here is replaced, because it yields a measurement this
repository does not otherwise have. Not whether tests cover the rules, which
measures an implementation, but whether two independent readings of the same
normative text produce the same rules, which measures the specification. Agreement
means the text is doing its job; divergence names the sentence that is missing.

§7.1 was already that measurement run once and was not labelled as one. Two
implementations written without reference to this document agree with §4.1 and
with each other on the digest preimage, and split on the unresolvable-algorithm
question -- one calling it unreadable, one calling it tampering. The first result
is the text working. The second is a located gap that took no argument to find,
because the same question was put to two implementations rather than debated.

It also settles what a second profile design would be for. One reading measures
nothing, so an independently written set of binding rules is the experiment, not
a collision with this one.

No rule, vector or suite changed. 585 passed.

* rfc: correct what the roadmap's "mutual case" refers to

The previous entry read that line as mutual delegation and reported this document
as short of it. That was a guess at the referent, made without checking, and it
is wrong.

In the reference implementation the mutual case is mutual attestation.
ca2a/docs/spec/mutual-attestation.md describes a callee-issued challenge and a
caller offer bound to it, so each side establishes what the other is running
before a payload opens. It separates the two concerns explicitly -- it
"establishes what each side is running", while "the delegation chain remains the
thing that says what it is allowed to ask for" -- and it does not mention a Trust
Record anywhere. Nor does cA2A's own docs/spec/trace-a2a-profile.md, whose A2A
profile is the delegation-link block and nothing else.

Which changes what the entry says. The roadmap asks the A2A profile to cover
something with no record representation today in either repository, sitting at
the transport layer rather than on this surface. That is a scoping question --
either mutual attestation gains a binding into the record, which is a schema
question rather than a verification one, or the v0.3 profile is two profiles --
and not a coverage failure in these rules. A reader comparing this document
against that roadmap line would otherwise conclude the second.

The bidirectional-delegation reading is kept as a separate note rather than
dropped, because it is true and unreachable for the reason section 4.2 gives, and
because the two readings should not merge later.

Every claim above traced to its file before writing: grep for "trace" in
mutual-attestation.md returns 0, grep for "mutual" in trace-a2a-profile.md
returns 0, and both quotations were checked against the source with whitespace
normalised, since the file wraps mid-sentence and a single-line grep misses them.

No rule, vector or suite changed. 585 passed, ruff clean.

* test(adequacy): grade delegation-link by the criteria this repository merged

#186 added criteria that every vector set on disk is measured against, and
test_every_vector_set_on_disk_is_measured_somewhere fails for a set in neither
SETS nor MEASURED_ELSEWHERE. This branch was opened three days before those
criteria landed, so merging upstream leaves `delegation-link` as the one set
nothing grades, and it is the only failure in the merged tree.

Registered in SETS rather than named in MEASURED_ELSEWHERE, because the set
holds up when it is actually graded rather than only pointed at:

    delegation-link: 23 vectors, 3 accepting, 10 distinct failure codes

No shortfall on either criterion decidable from the fixtures. It is not
satisfiable by an implementation that answers "accept" to everything or one
that answers "reject" to everything, and every one of the ten failure codes is
carried by exactly two vectors, which is the margin #124 asks for.

Boundaries are counted by failure code, the default. adequacy.py says that
assumption is the set's to justify: here the codes are the unit, because
tests/delegation_margins.json records the per-code margin and
tests/test_delegation_completeness.py holds each rule to being load-bearing for
both of its vectors, deleting the rule from the registry rather than matching
source text. The criteria adequacy.py leaves to each set, a rule nothing pins
and a weakness shared across a boundary's vectors, are implemented there too,
by rebuilding the registry without an entry and by substituting shortcut checks
that read only the first link or the first hop.

605 passed, 1 skipped. Verified by removing the SETS entry again, which fails
test_every_vector_set_on_disk_is_measured_somewhere on its own.

Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>

* rfc: the sha384 divergence is closed, so stop reporting it as live

Section 7.1 recorded the cA2A disagreement in the present tense: its block
validator accepts a sha384: link, compares it against a hash it only computes
as sha256:, and reports ProvenanceLinkBroken with "a tampered or reparented
record was detected". That was true when the corpus was run and is not true
now, so the document was carrying a defect report against another repository
that the other repository has already fixed.

Checked at ca2a 52141e8 rather than taken from the report:
src/ca2a_verify/dag.py:194 raises TraceDigestUnsupported with the detail "the
chain is unverifiable here, not invalid" at line 199, and the parent-link
comparison that produced ProvenanceLinkBroken is at line 201, after it. So the
guard precedes the comparison and vectors 22 and 23 now describe fixed
behaviour.

Both places are re-tensed rather than deleted. What the case establishes is not
that one verifier had a bug: a corpus written to argue a rule found the case,
the other implementation changed, and the shape it changed to is the
distinction section 4.3 asks for. Deleting it would drop the strongest evidence
in the document that the corpus does what it claims. The second passage said
two of three implementations distinguish unreadable from contradicted; it is
now all three.

Section 4.3's rule text is untouched, since it states the rule rather than
reporting on an implementation.

605 passed, 1 skipped.

Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>

---------

Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
lywinged added a commit to lywinged/trace-spec that referenced this pull request Aug 24, 2026
…ded references

Rebased onto upstream/main, 26 commits behind. Six files were flagged by the overlap
check and three needed a real merge; the directions are the ones DECISIONS.md recorded
on 2026-08-10, because the situation is the same one.

models.py and __init__.py are upstream's plus this fork's two profile constants and
the accepted_profiles machinery. sign.py merged cleanly as this fork's richer version,
superseding the minimal cutover check upstream took from agentrust-io#125, which CLAUDE.md section
5 predicted when agentrust-io#125 was offered.

The adequacy criteria are now upstream's, from agentrust-io#186, and the fork's own copy is dropped
rather than merged: upstream's went through review, carries the completeness guard the
fork's did not, and gained a delegation-link set from agentrust-io#184. Only the verifier-compatibility
loader is re-applied, since that set is fork-only, along with its recorded shortfall.
The pyproject pythonpath entry goes with it: it existed for a bare `from adequacy
import` that upstream's version does not use.

Upstream's models.py docstring says "TRACE v0.1 Trust Record" on a class whose
eat_profile is Literal["...trace-v0.2"]. The fork's correction is kept.

The crosswalk guard did what it exists to do. Upstream replaced "Verifiers MUST consult
current revocation status" with a SCITT-anchored model and added the references block,
and the source-derived inventory failed until nine rows moved. The four references rows
are mostly absences: an assurance-neutral block staying neutral is discharged by nothing
deriving a platform value from it, and two of the four MUST NOTs bind verifier behaviour
that no record shape can express, which the schema description already says.

The revocation rows are the ones worth reading. This library implements the fallback in
3.2.3 and not the primary rule: there is no last_valid_entry_id, no log_id and no receipt
comparison in src/, so a record is rejected on a key match whether or not it carries an
anchor. Section 3.2.3 names that behaviour itself for deployments with no receipts, so
it is incomplete rather than wrong, and the row says which.

856 passed, 1 skipped; ruff and mypy clean. Each new row and the restored shortfall
entry were checked by mutation: deleting a row, misspelling an anchor, and dropping a
recorded boundary each turn the naming test red.

Signed-off-by: lywinged <louie.lunz@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:UNKNOWN Contributor check flagged UNKNOWN risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants