Repository navigation
fix(bench): move the store guards ahead of the output file, again - #1733
Merged
Merged
Conversation
The guards added in #1732 sat after the point where each script creates its output, which is the defect they were modelled on and which #1729 had already fixed once in `bench_decode.sh`. `bench_na_attention.sh` truncates `$OUTPUT` and writes a header before reaching its guard; `bench_longprompt.sh` runs `: > "$OUTPUT"` thirty lines before its own; `bench_all_models.sh` takes `$1` as the log name and starts writing to it immediately, so a stray argument created a file named after it. Running the guards during review produced exactly that: a file named `all` and an empty `na_attention_m5max_2026-09-10.csv`, both from invocations the guard then refused. All three now run the check before the output path is chosen, so nothing has been created or truncated when they refuse. The test is the one that would have caught it: an existing file named with `--output` has to survive a refused run. All three exit 1 with the file byte-identical, and no stray output is left behind. Checking only that the guard fires is what let this through the first time, because a guard that fires after the damage still fires.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1732. The three guards it added sit after the point where each script creates its output, so they refuse only once the damage is done.
bench_na_attention.shtruncates$OUTPUTand writes a header before reaching the guard.bench_longprompt.shruns: > "$OUTPUT"thirty lines before its own.bench_all_models.shtakes$1as the log filename and writes to it immediately, so a stray argument creates a file named after it.This is the same defect #1729 fixed in
bench_decode.sh, reintroduced two PRs later. It surfaced because running the guards during review left a file calledalland an emptyna_attention_m5max_2026-09-10.csvin the tree, both from invocations that had been correctly refused.All three now check before the output path is chosen. The test is the one that distinguishes the two versions: an existing file named with
--outputhas to survive a refused run. All three exit 1 with the file byte-identical and leave nothing behind. Verifying that the guard fires is not enough, because a guard that fires after the truncation still fires, and that is what passed review the first time.