Skip to content

silent-revert canary: clean rows carry no attribution assertion, so their figures rot silently #2879

Description

@kyle-sexton

Surfaced by fresh-context verification during #2847's calibration re-measurement.
Independent of #2847 and of #2865 — see the boundary note at the bottom.

The gap

scripts/silent-revert-incidents.txt rows carry an optional bracketed
attribution field, [<culprit-sha>=<lines>,...], and #2833 made the replay
assert it exactly: same culprits, same per-culprit counts, no extras, no
omissions. That field is what turned a fires row from a claim into an
assertion.

clean rows cannot carry one. The grammar deliberately rejects it:

# An attribution on a `clean` row is nonsense -- clean rows have no findings.

and the replay exits 2 if one appears (t_replay_asserts_the_recorded_attribution
covers this). That is defensible on its own terms — a clean row has no finding
above the threshold — but it leaves the row's quantitative claim unchecked by
anything, while the prose around it treats that number as measured fact.

The evidence that this is not theoretical

Pinning the detector's diff flags (#2837) moved two figures. One was caught by
CI, one rotted silently, and the difference is exactly this gap:

row type figure outcome
fires cc58cbc53 [... eda5ae5ed=301] 301 → 298 caught — replay went red, figure corrected in the same PR
clean c8470efd0 ... 129 lines 129 → 136 rotted silently — nothing asserts it; still says 129

Same root cause, same PR, same corpus. The fires row was protected because
its number is asserted; the clean row was not because its number cannot be.

Measured under the pinned flags (--diff-algorithm=myers -M,
--no-ignore-revs-file), ack file emptied so the raw attribution is exposed:

c8470efd09d78d924a962483ff8a0b7809180e11
  6370a44e75986a12ec2a79a2e8deec39558f6f53 136

The row still passes — 136 is below the 200 threshold, so the commit stays
clean and the binary expectation holds. It is only the number in the note that
is wrong.

Why the number matters

The clean row is the corpus's false-positive control, and the file explains
that it was chosen deliberately:

the closest sub-threshold miss found in 500 commits of real history, since
that is the row that breaks first if someone lowers the threshold to chase
recall

That is a quantitative, superlative claim resting entirely on the unasserted
number. So the one figure the calibration argument leans on hardest is the one
figure CI cannot check. (Whether this row is actually the closest miss is a
separate defect — #2865 — and the answer there is no.)

Scope: this recurs

Nothing about the gap is specific to c8470efd0. Any future clean row will
carry an unasserted line count in its note, and the next change to the diff or
blame flags will rot it the same way, silently, while the row keeps passing.

Possible shapes (not prescribing)

  1. A distinct field asserting the row's largest sub-threshold attribution —
    checks the number without changing the row's pass/fail semantics, and would
    have caught 129 → 136.
  2. Assert only the property actually claimed — that this row is the maximum
    sub-threshold finding in the corpus — which is stronger and would also have
    caught silent-revert canary: the pinned clean row is not the closest sub-threshold miss, and its note names the wrong PR #2865's defect.
  3. Keep clean rows unasserted but forbid quantitative claims in their notes, so
    nothing unverifiable is written down.

Option 2 is the most faithful to why the row exists, and the most expensive.

Boundary with #2865

Issue #2865 is about this row's content being wrong: it names the wrong PR
(#2679 vs the measured #2715) and asserts a superlative that is false (the
closest sub-threshold miss is 195, not 129/136). Fixing that corrects this
row, once
.

This issue is about the absence of any assertion on clean rows — a
structural gap that lets the next clean row rot the same way after #2865 is
closed. Different fix, different lifetime; filed separately on purpose.

Not blocking

PR #2847 is re-deriving the calibration prose and does not depend on this.
Deliberately not folded into it.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions