Repository navigation
fix(coverage-report): the mutation tooling globbed the corpus flat - #4
Merged
Merged
Conversation
`mutation_report.py` discovered vectors with `FIXTURE_DIR.glob("*.json")`
and saw thirty of forty-eight. Everything one directory down, in
`conformance/proposal-117/`, was invisible to it, and `pair_mutation` and
`triple_mutation` import `FIXTURES` from it, so all three ran on a corpus
missing a fifth of itself.
That is not a miscount, it changed the answers:
- `receipt_gap_disclosed` is held by exactly two vectors and both are one
directory down. The script reported the obligation as held by nothing
and exited non-zero. On the full corpus it is held, with a full-outcome
margin of 2, which is the margin agentrust-io#124 asks for.
- `pair_mutation` and `triple_mutation` reported "no rule is masked by any
other" and "nothing new at rank three" and exited zero while doing it.
Those conclusions still hold on all forty-eight, measured. They were
correct and unearned; now they are earned.
The second case is the worse one. `check_canonicalizer.py`, in this same
directory, refuses to run rather than report success over a corpus it
cannot find, and says why: a differential check that reports success over
an empty corpus is the failure mode it exists to catch. Three scripts beside
it fell to exactly that.
This is the defect of agentrust-io#208, which upstream closed for
its two readers with a shared `discover_fixtures`. Importing that same
function rather than restating the walk is what stops a third reader from
drifting away from them again.
Scope: the flat glob only. `check_canonicalizer.py` reports four vectors
whose signing key is unpinned where no key failure is expected; that is a
separate question about the corpus, not about discovery, and it is not
touched here.
935 passed, 1 skipped. ruff clean. All three scripts exit 0 and report 48.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
|
❔ Contributor Check: UNKNOWN
Automated check by AgenTrust Contributor Check. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why these files belong in this change
One file changes:
coverage-report/scripts/mutation_report.py, one line of discovery plus the comment explaining it.pair_mutation.pyandtriple_mutation.pyare affected but not edited — theyfrom mutation_report import FIXTURES, so fixing the source fixes all three. Nothing else is in scope.The defect
mutation_report.py:49discovered vectors withFIXTURE_DIR.glob("*.json")— flat — and saw 30 of 48. Everything one directory down, inconformance/proposal-117/, was invisible.It changed the answers, in two different ways
Loudly.
receipt_gap_disclosedis held by exactly two vectors, both one directory down. The script reported the obligation as held by nothing and exited 1. On the full corpus:Held, with a full-outcome margin of 2, which is the margin agentrust-io#124 asks for. The script now also names why a status-only criterion would still call it unheld:
detectable ONLY with the finer oracle.Quietly, which is worse.
pair_mutationandtriple_mutationreported "no rule is masked by any other" and "nothing new at rank three" and exited 0 while computing them over a corpus missing a fifth of itself. Measured on all 48: both conclusions still hold. They were correct and unearned. Now they are earned.check_canonicalizer.py, in this same directory, refuses to run rather than report success over a corpus it cannot find, and says why:Three scripts beside it fell to exactly that.
The fix
Import
discover_fixturesfrom the verifier module — already onsys.path— rather than restating the walk. This is the defect of agentrust-io/trace-spec#208, which upstream closed for its two readers with that shared function. Sharing it is what stops a third reader from drifting away from them again.Verified
vectors: 48pairandtripleconclusions re-measured on the full corpus, unchangedDeliberately not in this PR
check_canonicalizer.pyreports four vectors whose signing key is unpinned where no key or signature failure is expected:Those are all "unknown key" / "key case variant" vectors, and
DECISIONS.md(2026-08-07) records that an unknown issuer key now yields unverified, not invalid — so the likely reading is that the script's expectation model predates that decision. That is a question about the corpus and the model, not about discovery, and it gets its own PR.Generated by Claude Code