Skip to content

fix(coverage-report): the mutation tooling globbed the corpus flat - #4

Merged
lywinged merged 1 commit into
mainfrom
fix/mutation-tooling-flat-glob
Aug 27, 2026
Merged

lywinged merged 1 commit into
mainfrom
fix/mutation-tooling-flat-glob

Conversation

@lywinged

Copy link
Copy Markdown
Owner

Why these files belong in this change

One file changes: coverage-report/scripts/mutation_report.py, one line of discovery plus the comment explaining it. pair_mutation.py and triple_mutation.py are affected but not edited — they from mutation_report import FIXTURES, so fixing the source fixes all three. Nothing else is in scope.

The defect

mutation_report.py:49 discovered vectors with FIXTURE_DIR.glob("*.json") — flat — and saw 30 of 48. Everything one directory down, in conformance/proposal-117/, was invisible.

扁平 glob: 30 条    递归: 48 条    差: 18
receipt_gap_disclosed 由 2 条向量持有 —— 两条都在 proposal-117/
扁平 glob 看到其中 0 条

It changed the answers, in two different ways

Loudly. receipt_gap_disclosed is held by exactly two vectors, both one directory down. The script reported the obligation as held by nothing and exited 1. On the full corpus:

receipt_gap_disclosed   status=0   full=2   attributed=yes
[margins] full outcome: min=2 max=2 mean=2.00
[load-bearing] every obligation is held by at least one vector

Held, with a full-outcome margin of 2, which is the margin agentrust-io#124 asks for. The script now also names why a status-only criterion would still call it unheld: detectable ONLY with the finer oracle.

Quietly, which is worse. pair_mutation and triple_mutation reported "no rule is masked by any other" and "nothing new at rank three" and exited 0 while computing them over a corpus missing a fifth of itself. Measured on all 48: both conclusions still hold. They were correct and unearned. Now they are earned.

check_canonicalizer.py, in this same directory, refuses to run rather than report success over a corpus it cannot find, and says why:

a differential check that reports success over an empty corpus is the failure mode this script exists to catch

Three scripts beside it fell to exactly that.

The fix

Import discover_fixtures from the verifier module — already on sys.path — rather than restating the walk. This is the defect of agentrust-io/trace-spec#208, which upstream closed for its two readers with that shared function. Sharing it is what stops a third reader from drifting away from them again.

Verified

  • All three scripts exit 0 and report vectors: 48
  • pair and triple conclusions re-measured on the full corpus, unchanged
  • 935 passed, 1 skipped; ruff clean

Deliberately not in this PR

check_canonicalizer.py reports four vectors whose signing key is unpinned where no key or signature failure is expected:

14-receipt-issuer-key-unknown.json
23-receipt-issuer-key-case-variant.json
07-gap-disclosure-unknown-key.json
09-gap-disclosure-key-case-variant.json

Those are all "unknown key" / "key case variant" vectors, and DECISIONS.md (2026-08-07) records that an unknown issuer key now yields unverified, not invalid — so the likely reading is that the script's expectation model predates that decision. That is a question about the corpus and the model, not about discovery, and it gets its own PR.


Generated by Claude Code

`mutation_report.py` discovered vectors with `FIXTURE_DIR.glob("*.json")`
and saw thirty of forty-eight. Everything one directory down, in
`conformance/proposal-117/`, was invisible to it, and `pair_mutation` and
`triple_mutation` import `FIXTURES` from it, so all three ran on a corpus
missing a fifth of itself.

That is not a miscount, it changed the answers:

- `receipt_gap_disclosed` is held by exactly two vectors and both are one
  directory down. The script reported the obligation as held by nothing
  and exited non-zero. On the full corpus it is held, with a full-outcome
  margin of 2, which is the margin agentrust-io#124 asks for.

- `pair_mutation` and `triple_mutation` reported "no rule is masked by any
  other" and "nothing new at rank three" and exited zero while doing it.
  Those conclusions still hold on all forty-eight, measured. They were
  correct and unearned; now they are earned.

The second case is the worse one. `check_canonicalizer.py`, in this same
directory, refuses to run rather than report success over a corpus it
cannot find, and says why: a differential check that reports success over
an empty corpus is the failure mode it exists to catch. Three scripts beside
it fell to exactly that.

This is the defect of agentrust-io#208, which upstream closed for
its two readers with a shared `discover_fixtures`. Importing that same
function rather than restating the walk is what stops a third reader from
drifting away from them again.

Scope: the flat glob only. `check_canonicalizer.py` reports four vectors
whose signing key is unpinned where no key failure is expected; that is a
separate question about the corpus, not about discovery, and it is not
touched here.

935 passed, 1 skipped. ruff clean. All three scripts exit 0 and report 48.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: lywinged <48041247+lywinged@users.noreply.github.com>
@github-actions

github-actions Bot commented Aug 26, 2026 •

Copy link
Copy Markdown

❔ Contributor Check: UNKNOWN

Check Result
Profile UNKNOWN
Credential LOW
Overall UNKNOWN

Automated check by AgenTrust Contributor Check.

@github-actions github-actions Bot added the needs-review:UNKNOWN Contributor check flagged UNKNOWN risk label Aug 26, 2026
@lywinged
lywinged merged commit faff83d into main Aug 27, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:UNKNOWN Contributor check flagged UNKNOWN risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant