Repository navigation
test(ci): the agreeing direction, so a louder gate cannot pass (#2472) - #2491
Merged
Merged
Conversation
check_duplicate_agreement had two controls and neither required silence: both assert returncode == 1. A gate rewritten to report a split on a tree where every copy agrees satisfies both. Measured before the fix: two such mutations were caught ONLY by --self-check-drop, which exercises a different branch entirely. The primary control passed both. Coverage by a sibling's accident is not coverage. The new case plants agreement -- same fixture for both copies -- and asserts one behaviour, exit 0. Both mutations are now killed by the primary control and a no-op mutation stays silent. My own comment broke my own harness on the way: it quoted the branch verbatim, a text-replacing mutation hit the COMMENT instead of the code, the gate ran unmutated, and the case reported itself blind. It describes the branch now and says why. Refs #2472
…ate (#2472) A comment explaining a new silence case quoted its branch verbatim. A text-replacing mutation then hit the comment instead of the code, the gate ran unmutated, and the case reported itself blind. str.replace takes the first occurrence and prose comes before code. Describe the branch instead, or mutate by line number. Plus: applying section 28 across all thirteen controls found one gate whose two controls both demand red, so a gate that reports a split on agreeing copies satisfies both. It was caught only by a sibling control written for something else -- coverage by accident, which a green report does not distinguish from the real thing. And: the sweep was a regex proxy over twelve files and a reading of one. The write-up says so, because a sweep reported without that sentence reads as twelve clean results. Refs #2472
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Yesterday's §28 was: a control made entirely of cases demanding a failure is blind to every mutation that makes a gate louder. Applied across all thirteen controls, one gate answered.
check_duplicate_agreementhad two controls and neither required silenceBoth assert
returncode == 1. A gate rewritten to report a split on a tree where every copy agrees satisfies both.Measured before the fix — two such mutations of the one-group branch:
Caught only by
--self-check-drop, which exists to exercise a different branch entirely. Coverage by a sibling's accident is not coverage, and nothing in a green report distinguishes the two.The new case plants agreement
Same fixture for both copies, so they genuinely agree; it asserts one behaviour and exit 0.
My own comment broke my own harness
The first version of that comment quoted the branch verbatim.
str.replace(needle, mut, 1)takes the first occurrence, prose comes before code, so the mutation hit the comment — the gate ran unmutated and the case reported itself blind. I nearly wrote up "the silence case does not work".It describes the branch now instead of quoting it, and says why. Settled by mutating by line number.
What the sweep did not do
The per-file count of silence-shaped strings is a proxy. One file was read and confirmed; the other twelve were scored by a regex and left there. Twelve unexamined counts is a different claim from twelve clean results, and the note says which this is.
Refs #2472