Repository navigation
feat(tri): gates mutate --loud, the direction six days of mutants never took (#2492) - #2493
Merged
Merged
Conversation
…er took (#2472) Every mutant this command made turned a failure into a pass, so it only ever asked "can this gate still fail?". A control made entirely of cases demanding RED satisfies all of them, and the opposite defect -- a gate that fails on a clean tree -- went unmeasured. --loud flips `return 0` to `return 1`. First run: seven of thirteen gates have survivors. Verified end to end on check_gate_preconditions.py, which is mine: OK: 9 precondition(s) across 6 gates fail loudly; 1 known-uncovered exit 1 control: exit 0 -- nothing noticed The gate prints a clean bill of health and reports failure. Exact mirror of this campaign's first finding, and it survived every iteration since because every mutation until today pushed the same way. The survivors are one class: four are the success return of --update-baseline, right after "baseline written: N entries". Nothing exercises the ledger-writing path's exit code. check_vector_data is the exception, and only because a case added three days ago for an unrelated reason happens to run --update-baseline. Only a BARE `return 0` counts. A ternary can yield 0 on one arm and a verdict on the other, so forcing the whole line to 1 is the silent operator seen backwards; a test asserts the two directions never claim the same line. Not fixed here: eight sites across seven gates is separate work. Fixing them in the change that introduced the tool that found them would leave nobody able to tell which was doing the work. Refs #2472
…res one way (#2492) For six days every mutant this campaign made turned a failure into a pass, so a control built entirely of cases demanding RED satisfied all of them. --loud asks the other question and found survivors in seven of thirteen gates -- the worst being the file that enforces this discipline for six others, which prints a clean bill of health and exits 1 with nothing noticing. The generalisation is not about mutation testing: any instrument that perturbs a system does so along some axis, and the axes you did not choose are invisible exactly the way a passing test is. Plus: note when a clean score is accidental, and land the tool with the finding rather than with the fixes -- otherwise nobody can tell which was doing the work. Refs #2472, #2492
Contributor
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For six days every mutant this command made turned a failure into a pass. Every one asked the same question — can this gate still fail? — so a control made entirely of cases demanding RED satisfied all of them, and the opposite defect went unmeasured across thirteen gates.
--loudreturn 0→return 1: does anything notice a gate that fails on a clean tree?First run, seven of thirteen gates have survivors — #2492 carries the table and the class.
Verified end to end on the worst one, which is mine
check_gate_preconditions.py— the file that enforces this discipline for six other gates:It prints a clean bill of health and reports failure. Exact mirror of this campaign's first finding —
check_catalog_integrityprintingOKwhile exiting 0 — and it survived every iteration since, because the tool and the blind spot shared a direction.Only a bare
return 0countsA ternary can yield 0 on one arm and a verdict on the other, so forcing the whole line to 1 is the silent operator seen backwards and would be scored against the wrong control. A test asserts the two directions never claim the same line, and that
return 1,return 0 if ok else 1andreturn failsare all excluded.Not fixed here
Eight sites across seven gates is separate work. Fixing them in the change that introduced the tool that found them would leave nobody able to tell whether the tool or the fixes were doing the work — and the tool is the part that has to survive being wrong.
Refs #2472, #2492