fix(coverage-report): the mutation tooling could not see the rule registry - #5
Merged
Merged
Conversation
…istry
The verifier's obligations moved into an explicit `RULES` registry. Site
discovery in `mutation_report` looks for two literal forms, `failures.append("code")`
and an inline `failures=[...]` argument, and the registry is neither. There are
now zero append-form sites in the verifier, so discovery found exactly one site,
an inline warning that is not a registry rule at all, and all twenty-two declared
obligations were invisible to every mutation script.
None of the three said so. Measured on the tree before this change:
mutation_report sites: 1 "every obligation is held by at least one vector" exit 0
pair_mutation pairs: 0 "no rule is masked by any other" exit 0
triple_mutation mutants: 0 "Nothing new at rank three." exit 0
Zero mutants evaluated, three affirmations, three clean exits. That is the same
output a suite with no checks in it produces, and nothing in it distinguishes the
two cases.
Three changes.
Discovery learns the registry form. A `Rule("code", "severity", ...)` element of
the `RULES` tuple is a site, and neutralising it removes that element. Sites go
from 1 to 23, and all twenty-two registry obligations are now mutated: every one
comes back at full-outcome margin 2 or better and attributed to a vector that
names it, so the corpus was sound and only the measurement was blind. Second and
third order go from 0 mutants to 253 pairs and 1771 triples, and both hold.
The transformer is shared instead of copied. `mutation_report`, `pair_mutation`
and `triple_mutation` each carried their own identical `Drop`. Three copies is how
a newly discovered form gets applied by some readers and not others, and a form
that is discovered but not applied yields a mutant identical to the baseline,
which reads as "no vector notices" and scores the rule unheld for a reason that is
not about vectors. One `drop_sites`, imported by all three, the same shape as the
shared `discover_fixtures` that closed agentrust-io#208 and #4.
The scripts refuse rather than report when they cannot see the inventory.
`reconcile_or_refuse` compares discovered sites against the codes the verifier
declares in `RULES` and exits 2, distinct from the exit 1 a real coverage gap
earns, listing what it could not reach. `mutation_report` also refuses on zero
sites, `pair_mutation` on fewer than two, `triple_mutation` on fewer than three.
A verifier with no registry declares no inventory and the reconciliation passes
vacuously there, which is the limit of the guard and is documented as such.
Each guard was tested by causing the condition it exists to catch, because a
guard that has never fired is the defect it is meant to prevent. Blinding registry
discovery returns all three to exit 2 with "22 of 22 declared obligations are
invisible to site discovery" where they previously exited 0 affirming; blinding
the declaration as well leaves pair and triple refusing on the count; blinding
every form leaves mutation_report refusing on zero sites.
935 passed, 1 skipped. Ruff reports 11 findings on this directory both before and
after, none of them new.
Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
|
❔ Contributor Check: UNKNOWN
Automated check by AgenTrust Contributor Check. |
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while verifying #4. #4 is correct and merged; it is what made this visible, by getting the three scripts far enough to show what they were actually measuring.
The defect
The verifier's obligations live in an explicit
RULESregistry. Site discovery inmutation_reportlooks for two literal forms,failures.append("code")and an inlinefailures=[...]argument. The registry is neither, and there are now zero append-form sites in the verifier. Discovery found exactly one site, an inline warning that is not a registry rule at all, so all twenty-two declared obligations were invisible to every mutation script.None of the three scripts said so. Measured on the tree before this change:
Zero mutants evaluated, three affirmations, three clean exits. That is the same output a suite with no checks in it produces, and nothing in the output distinguishes the two cases.
Three changes
Discovery learns the registry form. A
Rule("code", "severity", ...)element of theRULEStuple is a site, and neutralising it removes that element. Sites go from 1 to 23. All twenty-two registry obligations now come back at full-outcome margin 2 or better and attributed to a vector that names them, so the corpus was sound and only the measurement was blind. Second and third order go from 0 mutants to 253 pairs and 1771 triples, and both hold.The transformer is shared instead of copied. All three scripts carried their own identical
Drop. Three copies is how a newly discovered form gets applied by some readers and not others, and a form that is discovered but not applied yields a mutant identical to the baseline, which reads as "no vector notices" and scores the rule unheld for a reason that has nothing to do with vectors. Onedrop_sites, imported by all three, the same shape as the shareddiscover_fixturesthat closed agentrust-io#208 and #4.The scripts refuse rather than report when they cannot see the inventory.
reconcile_or_refusecompares discovered sites against the codes the verifier declares inRULES, and exits 2 (distinct from the exit 1 a real coverage gap earns) listing what it could not reach.mutation_reportalso refuses on zero sites,pair_mutationon fewer than two,triple_mutationon fewer than three.A verifier with no registry declares no inventory, so the reconciliation passes vacuously on such a tree. That is the limit of the guard rather than an oversight, and it is documented in the docstring: partial blindness is only detectable against a declaration of what should have been found, which is the argument for the registry existing.
Every guard was tested by causing the condition it exists to catch
A guard that has never fired is the defect it is meant to prevent, so each was made to fire.
22 of 22 declared obligations are invisible to site discovery— where before they exited 0 affirmingpair_mutationandtriple_mutationexit 2 on the count guard; this also confirms the reconciliation does not break a registry-less treemutation_reportexits 2 on the zero-site guardThe third probe was added because the second one found a hole:
mutation_reporthad no count guard and still exited 0 affirming over a single site.Checks
935 passed, 1 skippedunder.venv. Ruff reports 11 findings on this directory both before and after, none of them new — the one I introduced (Sequenceimported fromtyping) is fixed tocollections.abc.Reproduce:
Generated by Claude Code