Skip to content

fix(coverage-report): the mutation tooling could not see the rule registry - #5

Merged
lywinged merged 1 commit into
mainfrom
fix/mutation-tooling-registry-blind
Aug 27, 2026
Merged

lywinged merged 1 commit into
mainfrom
fix/mutation-tooling-registry-blind

Conversation

@lywinged

Copy link
Copy Markdown
Owner

Found while verifying #4. #4 is correct and merged; it is what made this visible, by getting the three scripts far enough to show what they were actually measuring.

The defect

The verifier's obligations live in an explicit RULES registry. Site discovery in mutation_report looks for two literal forms, failures.append("code") and an inline failures=[...] argument. The registry is neither, and there are now zero append-form sites in the verifier. Discovery found exactly one site, an inline warning that is not a registry rule at all, so all twenty-two declared obligations were invisible to every mutation script.

registry obligations sites found obligations mutated
before 22 1 0
after 22 23 22

None of the three scripts said so. Measured on the tree before this change:

mutation_report   sites: 1     "every obligation is held by at least one vector"   exit 0
pair_mutation     pairs: 0     "no rule is masked by any other"                    exit 0
triple_mutation   mutants: 0   "Nothing new at rank three."                        exit 0

Zero mutants evaluated, three affirmations, three clean exits. That is the same output a suite with no checks in it produces, and nothing in the output distinguishes the two cases.

Three changes

Discovery learns the registry form. A Rule("code", "severity", ...) element of the RULES tuple is a site, and neutralising it removes that element. Sites go from 1 to 23. All twenty-two registry obligations now come back at full-outcome margin 2 or better and attributed to a vector that names them, so the corpus was sound and only the measurement was blind. Second and third order go from 0 mutants to 253 pairs and 1771 triples, and both hold.

The transformer is shared instead of copied. All three scripts carried their own identical Drop. Three copies is how a newly discovered form gets applied by some readers and not others, and a form that is discovered but not applied yields a mutant identical to the baseline, which reads as "no vector notices" and scores the rule unheld for a reason that has nothing to do with vectors. One drop_sites, imported by all three, the same shape as the shared discover_fixtures that closed agentrust-io#208 and #4.

The scripts refuse rather than report when they cannot see the inventory. reconcile_or_refuse compares discovered sites against the codes the verifier declares in RULES, and exits 2 (distinct from the exit 1 a real coverage gap earns) listing what it could not reach. mutation_report also refuses on zero sites, pair_mutation on fewer than two, triple_mutation on fewer than three.

A verifier with no registry declares no inventory, so the reconciliation passes vacuously on such a tree. That is the limit of the guard rather than an oversight, and it is documented in the docstring: partial blindness is only detectable against a declaration of what should have been found, which is the argument for the registry existing.

Every guard was tested by causing the condition it exists to catch

A guard that has never fired is the defect it is meant to prevent, so each was made to fire.

probe result
registry discovery blinded all three exit 2, 22 of 22 declared obligations are invisible to site discovery — where before they exited 0 affirming
discovery and declaration blinded (the pre-refactor shape) pair_mutation and triple_mutation exit 2 on the count guard; this also confirms the reconciliation does not break a registry-less tree
every form blinded, zero sites mutation_report exits 2 on the zero-site guard

The third probe was added because the second one found a hole: mutation_report had no count guard and still exited 0 affirming over a single site.

Checks

935 passed, 1 skipped under .venv. Ruff reports 11 findings on this directory both before and after, none of them new — the one I introduced (Sequence imported from typing) is fixed to collections.abc.

Reproduce:

python coverage-report/scripts/mutation_report.py  "$PWD"
python coverage-report/scripts/pair_mutation.py    "$PWD"
python coverage-report/scripts/triple_mutation.py  "$PWD"

Generated by Claude Code

…istry

The verifier's obligations moved into an explicit `RULES` registry. Site
discovery in `mutation_report` looks for two literal forms, `failures.append("code")`
and an inline `failures=[...]` argument, and the registry is neither. There are
now zero append-form sites in the verifier, so discovery found exactly one site,
an inline warning that is not a registry rule at all, and all twenty-two declared
obligations were invisible to every mutation script.

None of the three said so. Measured on the tree before this change:

    mutation_report   sites: 1     "every obligation is held by at least one vector"   exit 0
    pair_mutation     pairs: 0     "no rule is masked by any other"                    exit 0
    triple_mutation   mutants: 0   "Nothing new at rank three."                        exit 0

Zero mutants evaluated, three affirmations, three clean exits. That is the same
output a suite with no checks in it produces, and nothing in it distinguishes the
two cases.

Three changes.

Discovery learns the registry form. A `Rule("code", "severity", ...)` element of
the `RULES` tuple is a site, and neutralising it removes that element. Sites go
from 1 to 23, and all twenty-two registry obligations are now mutated: every one
comes back at full-outcome margin 2 or better and attributed to a vector that
names it, so the corpus was sound and only the measurement was blind. Second and
third order go from 0 mutants to 253 pairs and 1771 triples, and both hold.

The transformer is shared instead of copied. `mutation_report`, `pair_mutation`
and `triple_mutation` each carried their own identical `Drop`. Three copies is how
a newly discovered form gets applied by some readers and not others, and a form
that is discovered but not applied yields a mutant identical to the baseline,
which reads as "no vector notices" and scores the rule unheld for a reason that is
not about vectors. One `drop_sites`, imported by all three, the same shape as the
shared `discover_fixtures` that closed agentrust-io#208 and #4.

The scripts refuse rather than report when they cannot see the inventory.
`reconcile_or_refuse` compares discovered sites against the codes the verifier
declares in `RULES` and exits 2, distinct from the exit 1 a real coverage gap
earns, listing what it could not reach. `mutation_report` also refuses on zero
sites, `pair_mutation` on fewer than two, `triple_mutation` on fewer than three.
A verifier with no registry declares no inventory and the reconciliation passes
vacuously there, which is the limit of the guard and is documented as such.

Each guard was tested by causing the condition it exists to catch, because a
guard that has never fired is the defect it is meant to prevent. Blinding registry
discovery returns all three to exit 2 with "22 of 22 declared obligations are
invisible to site discovery" where they previously exited 0 affirming; blinding
the declaration as well leaves pair and triple refusing on the count; blinding
every form leaves mutation_report refusing on zero sites.

935 passed, 1 skipped. Ruff reports 11 findings on this directory both before and
after, none of them new.

Signed-off-by: Louielunz <48041247+lywinged@users.noreply.github.com>
@github-actions

github-actions Bot commented Aug 27, 2026 •

Copy link
Copy Markdown

❔ Contributor Check: UNKNOWN

Check Result
Profile UNKNOWN
Credential LOW
Overall UNKNOWN

Automated check by AgenTrust Contributor Check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:UNKNOWN Contributor check flagged UNKNOWN risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant