You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Surfaced by fresh-context verification during #2847's calibration re-measurement.
Independent of #2847 and of #2865 — see the boundary note at the bottom.
The gap
scripts/silent-revert-incidents.txt rows carry an optional bracketed
attribution field, [<culprit-sha>=<lines>,...], and #2833 made the replay
assert it exactly: same culprits, same per-culprit counts, no extras, no
omissions. That field is what turned a fires row from a claim into an
assertion.
clean rows cannot carry one. The grammar deliberately rejects it:
# An attribution on a `clean` row is nonsense -- clean rows have no findings.
and the replay exits 2 if one appears (t_replay_asserts_the_recorded_attribution
covers this). That is defensible on its own terms — a clean row has no finding above the threshold — but it leaves the row's quantitative claim unchecked by
anything, while the prose around it treats that number as measured fact.
The evidence that this is not theoretical
Pinning the detector's diff flags (#2837) moved two figures. One was caught by
CI, one rotted silently, and the difference is exactly this gap:
row type
figure
outcome
fires cc58cbc53 [... eda5ae5ed=301]
301 → 298
caught — replay went red, figure corrected in the same PR
clean c8470efd0 ... 129 lines
129 → 136
rotted silently — nothing asserts it; still says 129
Same root cause, same PR, same corpus. The fires row was protected because
its number is asserted; the clean row was not because its number cannot be.
Measured under the pinned flags (--diff-algorithm=myers -M, --no-ignore-revs-file), ack file emptied so the raw attribution is exposed:
The row still passes — 136 is below the 200 threshold, so the commit stays
clean and the binary expectation holds. It is only the number in the note that
is wrong.
Why the number matters
The clean row is the corpus's false-positive control, and the file explains
that it was chosen deliberately:
the closest sub-threshold miss found in 500 commits of real history, since
that is the row that breaks first if someone lowers the threshold to chase
recall
That is a quantitative, superlative claim resting entirely on the unasserted
number. So the one figure the calibration argument leans on hardest is the one
figure CI cannot check. (Whether this row is actually the closest miss is a
separate defect — #2865 — and the answer there is no.)
Scope: this recurs
Nothing about the gap is specific to c8470efd0. Any future clean row will
carry an unasserted line count in its note, and the next change to the diff or
blame flags will rot it the same way, silently, while the row keeps passing.
Possible shapes (not prescribing)
A distinct field asserting the row's largest sub-threshold attribution —
checks the number without changing the row's pass/fail semantics, and would
have caught 129 → 136.
Issue #2865 is about this row's content being wrong: it names the wrong PR
(#2679 vs the measured #2715) and asserts a superlative that is false (the
closest sub-threshold miss is 195, not 129/136). Fixing that corrects this
row, once.
This issue is about the absence of any assertion on clean rows — a
structural gap that lets the next clean row rot the same way after #2865 is
closed. Different fix, different lifetime; filed separately on purpose.
Not blocking
PR #2847 is re-deriving the calibration prose and does not depend on this.
Deliberately not folded into it.
Surfaced by fresh-context verification during #2847's calibration re-measurement.
Independent of #2847 and of #2865 — see the boundary note at the bottom.
The gap
scripts/silent-revert-incidents.txtrows carry an optional bracketedattribution field,
[<culprit-sha>=<lines>,...], and #2833 made the replayassert it exactly: same culprits, same per-culprit counts, no extras, no
omissions. That field is what turned a
firesrow from a claim into anassertion.
cleanrows cannot carry one. The grammar deliberately rejects it:and the replay exits 2 if one appears (
t_replay_asserts_the_recorded_attributioncovers this). That is defensible on its own terms — a clean row has no finding
above the threshold — but it leaves the row's quantitative claim unchecked by
anything, while the prose around it treats that number as measured fact.
The evidence that this is not theoretical
Pinning the detector's diff flags (#2837) moved two figures. One was caught by
CI, one rotted silently, and the difference is exactly this gap:
fires cc58cbc53 [... eda5ae5ed=301]clean c8470efd0 ... 129 linesSame root cause, same PR, same corpus. The
firesrow was protected becauseits number is asserted; the
cleanrow was not because its number cannot be.Measured under the pinned flags (
--diff-algorithm=myers -M,--no-ignore-revs-file), ack file emptied so the raw attribution is exposed:The row still passes — 136 is below the 200 threshold, so the commit stays
clean and the binary expectation holds. It is only the number in the note that
is wrong.
Why the number matters
The
cleanrow is the corpus's false-positive control, and the file explainsthat it was chosen deliberately:
That is a quantitative, superlative claim resting entirely on the unasserted
number. So the one figure the calibration argument leans on hardest is the one
figure CI cannot check. (Whether this row is actually the closest miss is a
separate defect — #2865 — and the answer there is no.)
Scope: this recurs
Nothing about the gap is specific to
c8470efd0. Any futurecleanrow willcarry an unasserted line count in its note, and the next change to the diff or
blame flags will rot it the same way, silently, while the row keeps passing.
Possible shapes (not prescribing)
checks the number without changing the row's pass/fail semantics, and would
have caught 129 → 136.
sub-threshold finding in the corpus — which is stronger and would also have
caught silent-revert canary: the pinned clean row is not the closest sub-threshold miss, and its note names the wrong PR #2865's defect.
nothing unverifiable is written down.
Option 2 is the most faithful to why the row exists, and the most expensive.
Boundary with #2865
Issue #2865 is about this row's content being wrong: it names the wrong PR
(#2679 vs the measured #2715) and asserts a superlative that is false (the
closest sub-threshold miss is 195, not 129/136). Fixing that corrects this
row, once.
This issue is about the absence of any assertion on clean rows — a
structural gap that lets the next clean row rot the same way after #2865 is
closed. Different fix, different lifetime; filed separately on purpose.
Not blocking
PR #2847 is re-deriving the calibration prose and does not depend on this.
Deliberately not folded into it.