fix(scripts): give harness self-test an independent tally - #3285
Conversation
Source the library for assertions, but keep a shadow pass/fail count the harness cannot write. After test_harness::report, that tally forces a non-zero exit when any assertion failed — so a sabotaged return 1 or fail() increment cannot go green in CI. Add a throwaway-copy mutation regression for both defects. Co-authored-by: Kyle Sexton <kyle-sexton@users.noreply.github.com>
|
Claude finished @kyle-sexton's task in 42s —— View job Security review — no findings
Scope: Skip gate: the change has no security-relevant surface. It's a local, self-contained test file: no network calls, no secrets/credentials, no user-controlled input. The new mutation-regression helper ( No security issues found in this diff. |
|
Last security-reviewed head: |
|
Claude finished @kyle-sexton's task in 27s —— View job /review:code-review — skipped
This PR is already merged, which trips the If a fresh review is wanted (e.g. against a follow-up branch), please re-trigger |
|
Claude has reviewed this PR 1 time. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
Closes #3283
Summary
scripts/lib/test-harness.test.shsources the library it tests, so a break intest_harness::report's return path or infail()counter arithmetic is detected by the suite's assertions and then cannot be reported to CI. The process exit status is produced by the code under test.Two mutations against a throwaway copy of the harness demonstrate it: deleting
return 1from report, and removing the increment fromfail(). Both printFAIL:lines and still exited 0. Checking_test_harness_failafter report catches the first mutation but not the second.Fix
self_ok/self_fail, which keep an independent pass/fail tally the harness functions do not write.test_harness::report, a non-zero independent tally forces exit 1 even if the harness counters or return were sabotaged.test_harness::report; this self-test is the one file allowed to follow the call.Verification
bash scripts/lib/test-harness.test.sh— PASS=18 FAIL=0 (includes both mutation cases)bash scripts/check-silent-revert.test.sh— PASS=130 FAIL=0 (installs an EXIT trap and sources the harness)bash scripts/check-skill-portability.test.sh— PASS=92 FAIL=0 (installs an EXIT trap and sources the harness)shellcheck --rcfile=.shellcheckrc -xandshfmt -dclean onscripts/lib/test-harness.test.shbash -nclean onscripts/lib/test-harness.test.shRelated