Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions .claude/skills/ci-gates/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -985,3 +985,83 @@ backticks being command substitution, which cost text out of a commit message
and then out of a filed issue earlier in this campaign. **Long messages go
through `-F` or a quoted heredoc, every time — there is no version of this
that is worth retyping.**

## 26. A fix in the tool is not a fix at the terminal

A reviewer found that `tri gates mutate` never cleared Python's bytecode cache:
a `.pyc` is keyed on (source mtime in whole seconds, source size), `return 1` →
`return 0` preserves the size, and the loop writes well inside one second. That
was fixed in the tool.

Two days later I hit the identical defect **by hand**, on the same gate, while
checking somebody's recommendation. Five quick mutations left a stale `.pyc`,
and the control then reported five failures on a tree whose `git status` was
empty and whose source sha matched HEAD. I spent a round chasing a phantom.

**Fixing a hazard inside one tool leaves it live everywhere a person does the
same thing manually** — and the manual path is the one used while investigating,
which is exactly when a false red is most expensive.

The repair belongs in the artefact that can be reached both ways. Here it is
three lines in the control: drop the imported module's cached bytecode before
importing it.

### Every verdict taken before you found the broken instrument is suspect

Not wrong — **suspect**. The control could have been red for the right reason or
the wrong one, and nothing in the output distinguishes them. Re-run all of them
and say in the write-up that the numbers are a re-measurement. I did, and they
held; if they had not, publishing the first set would have been the finding.

### Refuting a recommendation needs the same rigour as making one

The recommendation was "the drift verdict is not covered; add a whole-process
case". Three mutations of increasing narrowness — the broadest, the wiring only,
and the drift path alone with bad-input untouched — plus a **no-op mutation as a
control on the control**. All three killed, the no-op silent. The
recommendation is refuted, and refuting it needed more measurements than
accepting it would have.

An unexamined recommendation costs an iteration of work. Accepting one because
the reviewer was right about four other things is how a review's authority
outlives its evidence.

## 27. Sweep for the class, not the instance

`tri gates mutate` was taught to clear Python's bytecode cache after a reviewer
found it there. I then hit the same defect by hand in a control. Fixing that one
control would have been the instance.

A grep for *files importing a sibling module* found **three**, and the one nobody
had tripped over was the worst:

```
wp18_selftest_gate.py imports wp18_conformance_gate
wp18_gate_selfconsistent_selftest.py imports wp18_conformance_gate
verify_exhaustive.py imports ternary_model
```

`verify_exhaustive` is the gate that proves four backends agree with a
**separately written** model. A stale model means the agreement is measured
against a version nobody is looking at — and that gate's whole value is that its
fourth opinion is independent.

**After fixing a defect, spend one grep on the shape of it.** The query is
usually mechanical: the language feature that made it possible (`^import (\w+)`
against the local module names), not the symptom. Two of the three had never
produced a wrong answer, which is why nobody had found them.

### Prove the fix on the instance you did not stumble into

The guard was verified on `verify_exhaustive`, not on the file where I hit the
bug: a same-size edit to the model gives exit 1, and reverting then running
immediately gives exit 0. Testing the repair only where you already saw the
symptom confirms the diagnosis you already had.

### A same-size edit is the whole hazard

`return 1` → `return 0`. `else 2` → `else 0`. `return acc` → `return aaa`. Every
mutation a mutation-tester makes is size-preserving by design, so a mutation
harness and this cache are natural enemies. If you are writing anything that
edits source in a loop, clear the cache in the loop — and remember the loop is
sometimes a person.
23 changes: 23 additions & 0 deletions docs/now/2026-08-24-the-same-trap-in-three-places.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# NOW — the same trap in three places (2026-08-24)

A reviewer found that `tri gates mutate` never cleared Python's bytecode cache. It was fixed there two days ago. Today I hit the identical defect by hand, and then a grep found it in two more files that nobody had run into yet.

- **How Python loses an edit.** A `.pyc` is keyed on (source mtime in **whole seconds**, source size). A one-line edit that preserves the size, followed by a run inside the same second, is served the previous state's bytecode. `return 1` → `return 0` preserves the size. So does `else 2` → `else 0`.

- **It cost a round.** Five hand mutations of `wp18_conformance_gate.py` in quick succession left a `.pyc` that made its control report **five failures on a tree whose `git status` was empty and whose source sha matched HEAD**. Every "mutant killed" verdict taken before that discovery was suspect — not wrong, suspect, because a control can be red for the right reason or the wrong one and the output does not distinguish them. All were re-run after clearing the cache; they held.

- **The sweep.** A grep for files importing a sibling module found three that matter, none of them guarded:

| file | imports | why it matters |
|---|---|---|
| `wp18_selftest_gate.py` | `wp18_conformance_gate` | the control reads a stale gate |
| `wp18_gate_selfconsistent_selftest.py` | `wp18_conformance_gate` | same, second control |
| `verify_exhaustive.py` | `ternary_model` | **the gate compares against a stale independent model** |

The third is the worst of the three. `verify_exhaustive` is the gate that proves four backends agree with a model written separately; a stale model means the comparison is against a version nobody is looking at.

- **Proven on the worst one, both directions.** Breaking `ternary_model.py` with a same-size edit → exit 1. Reverting and running immediately → exit 0. Without the guard the second run reads the broken model.

- **Where the repair belongs.** Fixing a hazard inside one tool leaves it live wherever a person does the same thing by hand — and the manual path is the one used *while investigating*, which is when a false red costs most. The three lines go in the artefact both paths reach.

Refs #2472
21 changes: 20 additions & 1 deletion tools/verify_exhaustive.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,26 @@

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import functools # noqa: E402
import ternary_model as MODEL # noqa: E402
# T96: drop any cached bytecode for ternary_model BEFORE importing it.
#
# Python keys a .pyc on (source mtime in whole seconds, source size). A one-line
# edit that preserves the size, followed by a run inside the same second, hands
# this file the PREVIOUS state's bytecode. Measured on wp18_selftest_gate.py:
# five hand mutations left a .pyc that produced five failures on a tree matching
# HEAD exactly. `tri gates mutate` clears this between mutants; a person at a
# terminal does not.
def _drop_stale_bytecode():
import pathlib
cache = pathlib.Path(__file__).resolve().parent / "__pycache__"
for p in cache.glob("ternary_model.*.pyc"):
try:
p.unlink()
except OSError:
pass


_drop_stale_bytecode()
import ternary_model as MODEL # noqa: E402 # noqa: E402

# dot27 is pure, so memoising it changes nothing about what the model computes and
# takes full_adder over its 16,777,216 inputs from minutes to under a minute.
Expand Down
21 changes: 20 additions & 1 deletion tools/wp18_gate_selfconsistent_selftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,26 @@

HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
import wp18_conformance_gate as gate # noqa: E402
# T96: drop any cached bytecode for wp18_conformance_gate BEFORE importing it.
#
# Python keys a .pyc on (source mtime in whole seconds, source size). A one-line
# edit that preserves the size, followed by a run inside the same second, hands
# this file the PREVIOUS state's bytecode. Measured on wp18_selftest_gate.py:
# five hand mutations left a .pyc that produced five failures on a tree matching
# HEAD exactly. `tri gates mutate` clears this between mutants; a person at a
# terminal does not.
def _drop_stale_bytecode():
import pathlib
cache = pathlib.Path(__file__).resolve().parent / "__pycache__"
for p in cache.glob("wp18_conformance_gate.*.pyc"):
try:
p.unlink()
except OSError:
pass


_drop_stale_bytecode()
import wp18_conformance_gate as gate # noqa: E402 # noqa: E402

PASS = 0
FAIL = 0
Expand Down
26 changes: 25 additions & 1 deletion tools/wp18_selftest_gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,31 @@
import sys
import tempfile

import wp18_conformance_gate as G
# T96: drop any cached bytecode for the gate BEFORE importing it.
#
# Python keys a .pyc on (source mtime in whole seconds, source size). Editing
# the gate's `return 1` to `return 0` preserves the size, and an edit-run-edit
# loop finishes well inside one second -- so this control can be handed the
# PREVIOUS state's bytecode and go red on a tree that matches git exactly.
#
# Measured here, on this file, while testing this gate: five hand mutations in
# quick succession left a .pyc that made the control report five failures with
# `git status` clean and the source sha matching HEAD. `tri gates mutate` clears
# this between mutants; a person at a terminal does not, and neither did this
# control. Three lines remove the whole class.
def _drop_stale_bytecode():
import pathlib
here = pathlib.Path(__file__).resolve().parent / "__pycache__"
for p in here.glob("wp18_conformance_gate.*.pyc"):
try:
p.unlink()
except OSError:
pass


_drop_stale_bytecode()

import wp18_conformance_gate as G # noqa: E402

# The gate file as a SCRIPT. `G.__file__` is the module this control already
# imported, so a mutant of the gate is the thing that gets run -- no --root
Expand Down
Loading