Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 38 additions & 42 deletions .claude/skills/ci-gates/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -2498,48 +2498,44 @@ cache means the next run finishes them rather than starting over. A partial tabl
reported as partial is a measurement; the same table reported as complete is the
thing this campaign is about.

## 63. Why that control killed nothing, and what fixing it cost

`verify_emit_bitexact.py` scored **0 killed of 17**. The reason, once the survivor
lines were read rather than counted:

- `sys.exit(0 if ok else 1)` survived **both** return operators
- every FAIL branch of the comparison — timeout, step count, mismatch, resource
count, synth error — survived inversion

**All three of my cases leave through `skip()` or `broken()`.** The gate's own
verdict was never observed at all. A control that covers only preconditions is a
control *for* preconditions, and I wrote those three knowing that class.

**The plant that reaches the verdict moves one arm only.** The Python side comes
from the interpreter `g.run()`; the Verilog is emitted from the microcode
`steps`, not from `run()`. Perturbing the interpreter makes the model disagree
with an RTL that is unchanged — which is what a real bit-exactness failure looks
like. Perturbing shared arithmetic instead would move both arms together and
plant nothing.

Two cases now: a clean tree exits 0 saying `RTL == model BIT-EXACT`, and a
perturbed model exits 1 naming the disagreeing step. Five cases in the file, both
directions of the verdict.

### And the cost is real, and worth stating plainly

The mutation loop runs a gate's **whole control per mutant**. This control now
spawns two ~45-second whole-program runs, so seventeen sites cost roughly half an
hour — the measurement timed out at ten minutes and the marker caught the
interrupted tree, cleanly, with nothing leaked.

**That is not a reason to make the control cheaper.** A control that exercises
only what is fast to exercise is how this gate got to 0/17 in the first place.
The tension is inherent: *a control worth having is expensive, and the mutation
loop pays that cost once per mutant.* The cache is the answer — measure once,
reuse until the gate or its control changes — and `[cached]` is what keeps a
reused row honest.

**The guard chain worked end to end for the first time.** Timeout → marker names
the gate → `git checkout -- tools/` → tree clean → committed diff contains only
the intended change. Three iterations ago the same sequence leaked two mutants
into a branch.
## 64. Two gates, one mistake, made twice by the same author

`verify_multitarget.py` scored **0 killed of 7** — and the survivor lines are
identical in shape to `verify_emit_bitexact.py`'s: the gate's own
`sys.exit(0 if ok else 1)` under both return operators, and all three comparison
FAIL branches under inversion.

Both controls cover the skip pair. **Both leave through `skip()` and never reach
`main()`.** Two gates, one mistake, and I made it twice — the second time three
iterations after writing down the first.

That is what a *class* looks like when you have not internalised it: the rule was
recorded, the next control was written the same way, and only a measurement
caught it.

**Both plants move one arm.** In `verify_multitarget`, `py_ref` reads the Python
model while C and Rust come from `t27c`; perturbing `py_ref` makes the model
disagree with backends that are unchanged. Perturbing the spec or the emitter
would move every arm together and plant nothing — the same distinction that made
the `verify_emit_bitexact` plant work, and the reason both are one-line edits in
a specific place rather than "break something".

verify_emit_bitexact.py 0/17 -> 13/17
verify_multitarget.py 0/7 -> 5/7

Only boundary survivors remain in both.

### And the honesty mechanism needed its own correction

Re-measuring produced a row with **two columns measured and three reused**,
labelled `[cached]` wholesale. Under-claiming rather than over-claiming — the
safe direction, and still wrong, because the entire point of the marker is that
a reader can tell which they are looking at.

Three states now: no marker when fresh, `[cached]` when every column is reused,
`[3 cached, 2 fresh]` when mixed. The mixed case had to be reached by marking
cache entries stale by hand, which is worth noting: **a state that cannot occur
naturally during testing is a state nobody has seen your code produce.**

## 65. Triaging fifty boundary survivors, and a pull request that got 7 of 35 checks

Expand Down
20 changes: 20 additions & 0 deletions docs/now/2026-08-23-two-gates-one-mistake.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# NOW — two gates, one mistake, made twice (2026-08-23)

`verify_multitarget.py` scored **0 killed of 7**, and its survivor lines are identical in shape to `verify_emit_bitexact.py`'s: the gate's own `sys.exit(0 if ok else 1)` under both return operators, and all three comparison FAIL branches under inversion.

- **Both controls cover the skip pair. Both leave through `skip()` and never reach `main()`.** Two gates, one mistake, and I made it twice — the second time three iterations after writing down the first.
- That is what a *class* looks like when you have not internalised it: the rule was recorded, the next control was written the same way, and only a measurement caught it.
- **Both plants move one arm.** `py_ref` reads the Python model while C and Rust come from `t27c`; perturbing it makes the model disagree with backends that are unchanged. Perturbing the spec or the emitter would move every arm together and plant nothing.

```
verify_emit_bitexact.py 0/17 -> 13/17
verify_multitarget.py 0/7 -> 5/7
```

Only boundary survivors remain in both.

## The honesty mechanism needed its own correction

Re-measuring produced a row with **two columns measured and three reused**, labelled `[cached]` wholesale — under-claiming rather than over-claiming, so the safe direction, and still wrong: the point of the marker is that a reader can tell.

Three states now: no marker when fresh, `[cached]` when every column is reused, `[3 cached, 2 fresh]` when mixed. The mixed case had to be reached by marking cache entries stale by hand — **a state that cannot occur naturally during testing is a state nobody has seen your code produce.**
63 changes: 61 additions & 2 deletions tools/verify_multitarget.py
Original file line number Diff line number Diff line change
Expand Up @@ -247,8 +247,67 @@ def case(label, args, want_rc, want, forbid):
"--require was given, so a missing prerequisite is a failure"],
["SKIP", "ALL TARGETS BIT-EXACT", "CROSS-TARGET MISMATCH"])

print(" self-check: both exits of the skip pair reached, each by its own message; "
f"the cross-target mismatch verdict is NOT covered here = {ok}")
# T131: the gate's OWN verdict, which the two cases above never reach. The
# five-operator table scored this gate 0 killed of 7 -- `sys.exit(0 if ok
# else 1)` survived both return operators and all three comparison FAIL
# branches survived inversion, because both cases leave through skip().
#
# Identical in shape to verify_emit_bitexact.py, which scored 0 of 17 for
# the same reason. Two gates, one mistake, made twice by the same author.
#
# The plant moves ONE arm: py_ref reads the Python model, while C and Rust
# come from t27c. Perturbing py_ref makes the model disagree with backends
# that are unchanged -- a real cross-target divergence. Perturbing the spec
# or the emitter would move every arm together and plant nothing.
def whole_program(label, edit, want, expect, absent):
nonlocal ok
import shutil as _sh
with tempfile.TemporaryDirectory() as td:
tools = os.path.join(td, "tools")
os.makedirs(tools)
for f in os.listdir(os.path.join(ROOT, "tools")):
if f.endswith(".py"):
src = open(os.path.join(ROOT, "tools", f), encoding="utf-8").read()
if f == os.path.basename(__file__) and edit:
before = src
src = edit(src)
assert src != before, f"{label}: the plant changed nothing"
open(os.path.join(tools, f), "w", encoding="utf-8").write(src)
for extra in ("target", "specs"):
s = os.path.join(ROOT, extra)
if os.path.exists(s):
os.symlink(s, os.path.join(td, extra))
r = subprocess.run(
[sys.executable, os.path.join(tools, os.path.basename(__file__))],
capture_output=True, text=True, cwd=ROOT)
out = r.stdout + r.stderr
missing = [s for s in expect if s not in out]
leaked = [s for s in absent if s in out]
good = r.returncode == want and not missing and not leaked
print(f" {label:<44} " + (f"exit {want}, right branch" if good
else "CONTROL FAILED"))
if not good:
ok = False
print(f" exit {r.returncode!r} (want {want!r})")
if missing:
print(f" the branch never said: {missing!r}")
if leaked:
print(f" neighbouring marker leaked: {leaked!r}")
print(f" said {out[-300:]!r}")

whole_program("a clean tree agrees across targets", None, 0,
["ALL TARGETS BIT-EXACT"],
["CROSS-TARGET MISMATCH", "FAIL", "SKIP", "Traceback"])

whole_program("a perturbed model diverges from both backends",
lambda s: s.replace(
" return [f(a, b) & 0xFFFFFFFF for a, b in pairs]",
" return [(f(a, b) ^ 1) & 0xFFFFFFFF for a, b in pairs]", 1),
1, ["!= model", "CROSS-TARGET MISMATCH"],
["ALL TARGETS BIT-EXACT", "SKIP", "Traceback"])

print(" self-check: the skip pair and the gate's own verdict in both "
f"directions = {ok}")
return 0 if ok else 1


Expand Down
Loading