Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .claude/skills/ci-gates/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -1103,3 +1103,37 @@ the control dies there and every later case silently never runs.

A guard nobody has seen fire is the same evidential state as a gate nobody has
seen go red. Both are one planted fault away from being real.

## 29. Do not quote the line you are about to mutate

The comment explaining a new silence case quoted the branch it was about,
verbatim. A text-replacing mutation harness then hit the **comment** instead of
the code: the gate ran unmutated, the case reported itself blind, and I nearly
wrote up "the silence case does not work".

`str.replace(needle, mut, 1)` takes the first occurrence in the file, and prose
usually comes before code.

**Describe the branch, do not quote it** — "the one-group branch rewritten to a
constant false" reads as well and cannot be hit. If a literal is genuinely
needed, mutate **by line number** rather than by text, which is what settled it
here.

### Applying yesterday's rule found exactly one gate, and reading found it

A count of "silence-shaped" strings per control flagged one of thirteen.
`check_duplicate_agreement` had two controls and **neither required silence** —
both assert `returncode == 1`, so a gate reporting a split on a tree where every
copy agrees satisfies both.

Measured before the fix: two such mutations were caught **only** by
`--self-check-drop`, a control written to exercise a different branch. The
primary control passed them. **Coverage by a sibling's accident is not
coverage**, and nothing in a green report distinguishes the two.

### Say what the sweep did not look at

The count is a proxy. One file was read and confirmed; the other twelve were
scored by a regex and left there. The write-up says so. A sweep reported without
that sentence reads as twelve clean results, and twelve unexamined counts is a
different claim.
17 changes: 17 additions & 0 deletions docs/now/2026-08-24-a-control-that-only-demands-red.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# NOW — a control that only demands red (2026-08-24)

Yesterday's lesson was that a control made entirely of cases demanding a failure is blind to every mutation that makes a gate **louder**. Applied to all thirteen, one gate answered.

- **`check_duplicate_agreement` had two controls and neither required silence.** Both assert `returncode == 1`. A gate rewritten to report a split on a tree where every copy agrees satisfies both.

- **Measured before the fix.** Two such mutations — the one-group branch replaced by a constant false, and by a comparison against zero — were **caught only by `--self-check-drop`**, which exists to exercise a different branch entirely. The primary control passed both. Coverage by a sibling's accident is not coverage.

- **The new case plants agreement.** Same fixture for both copies, so they genuinely agree, and it asserts the gate says *one behaviour* and exits 0. Both mutations are now killed by the primary control, and a no-op mutation stays silent.

- **My own comment broke my own harness.** The first version quoted the branch verbatim. A text-replacing mutation then hit the **comment** rather than the code, the gate ran unmutated, and the case reported itself blind. I nearly wrote that up as "the silence case does not work". The comment now describes the branch instead of quoting it, and says why.

That is the fourth time this campaign that a measurement was wrong before it was right, and the first where the thing that corrupted it was documentation I had just added.

- **The sweep itself was a proxy and I checked it.** Counting "silence-shaped" strings per file flagged one gate. Reading that file confirmed it; reading is what turned a regex hit into a finding. The other twelve were not investigated beyond the count, and this note says so rather than implying a clean sweep.

Refs #2472
29 changes: 28 additions & 1 deletion tools/check_duplicate_agreement.py
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,34 @@ def self_check():
split = "FAIL tmul" in r.stdout and "DIFFERENT behaviours" in r.stdout
print(f" self-check: planted divergence reported as a split = {split}")
print(f" self-check: the gate's exit code on it = {r.returncode} (want 1)")
return 0 if (split and r.returncode == 1) else 1
# T98: the AGREEING direction. Every assertion above demands RED, and a
# control made only of those is blind to every mutation that makes the gate
# LOUDER -- the one-group branch rewritten to a constant false reports a
# split on a tree where the copies agree, and every assertion above is still
# satisfied. Measured: two such mutations passed this control and were
# caught only by --self-check-drop, which exists for something else.
# Coverage by a sibling's accident is not coverage.
#
# The branch is described rather than quoted on purpose: the first version
# of this comment carried the line verbatim, and a text-replacing mutation
# harness then hit the COMMENT instead of the code -- reporting the case as
# blind when the gate had never been mutated at all.
#
# Same fixture for both copies, so they genuinely agree.
with tempfile.TemporaryDirectory() as td:
os.makedirs(os.path.join(td, "specs/ternary"))
os.makedirs(os.path.join(td, "target/release"))
os.symlink(t, os.path.join(td, "target/release/t27c"))
for tag in ("A", "B"):
with open(os.path.join(td, "specs/ternary/%s.t27" % tag.lower()), "w") as fh:
fh.write(FIXTURE.format(tag=tag, agree="1", differ="-1"))
q = subprocess.run([sys.executable, os.path.abspath(__file__)],
capture_output=True, text=True,
env={**os.environ, "T27_DUP_ROOT": td})
quiet = ("DIFFERENT behaviours" not in q.stdout) and q.returncode == 0
print(" self-check: agreeing copies reported as one behaviour = %s (exit %d, want 0)"
% (quiet, q.returncode))
return 0 if (split and r.returncode == 1 and quiet) else 1


DROP_FIXTURE = """module DupFixtureDrop
Expand Down
Loading