Repository navigation
fix(gates): a control for verify_trainer_c; the selector was right by accident (#2492) - #2524
Merged
Merged
Conversation
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two measurements this iteration, and the honest one is the boring one.
The assert blind spot costs nothing here
is_gate_by_propertyrecognisesreturn,sys.exitandraise SystemExit. A tool that fails only through an uncaughtAssertionErroris invisible to it, and I expected that to be costing coverage.Measured across every CI-invoked tool in
tools/: zero fail only that way — every one has an explicit or ternary exit as well. The hole is real and empty. Saying so is worth more than the fix I was about to write.But the selector has a false positive, and it fired
gft_backprop_microcode.pywas classified as a gate on the strength ofwhich is a class label from a classifier, not a verdict.
verdict_literalswas built to recognise return statements that carry verdicts; using it to answer "can this file exit non-zero" conflates a returned value with an exit code.The file is a gate anyway — sixteen
asserts — so the classification is right by accident, which is the least useful way to be right. Recorded rather than patched: narrowing the predicate risks the false negative, and the one file affected is already correctly in the list.A control for the last of the four verifiers
verify_trainer_c.py— the one whoseskip()every other copied — now scores 1/1 silent, 1/1 loud, 3/3 invert, with a planted divergence in its C arm and the clean direction asserted beside it.The plant scopes its edit to the text after
def run_c(, because the same plant infuzz_trainer.pyhitrun_modelthree functions earlier and a case then passed on a divergence planted where it was never meant to be.Not covered, and named: the per-architecture "C produced N of M steps" branch, which needs a C arm that truncates rather than one that disagrees.
Where the count stands
Gates with no control in any form: 1 — and it is the file the selector was right about by accident.
The rule
A predicate reused outside the question it was written for will be right often enough to look correct.
verdict_literalsanswers "is this return a verdict?"; it does not answer "can this program fail?", and the difference only shows up on a classifier that returns 1 for a class.Refs #2492