Skip to content

fix(test): surface the failing lines when an unrecognised runner fails - #2683

Open
guyoron1 wants to merge 1 commit into
rtk-ai:developfrom
guyoron1:fix/test-generic-runner-fallback
Open

guyoron1 wants to merge 1 commit into
rtk-ai:developfrom
guyoron1:fix/test-generic-runner-fallback

Conversation

@guyoron1

@guyoron1 guyoron1 commented Jun 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #2420

When rtk test <cmd> doesn't recognise the runner (make test, bazel test, a custom script), it fell back to OUTPUT (last 5 lines):, which hid the failure whenever it wasn't in the tail. #2420's script prints FAIL: test_login broke among 50 PASS: lines, and only the last five PASS: lines came out.

Now, only when nothing was recognised (no summary, no runner diagnostic block) and the exit code is non-zero:

  • A. the error blocks rtk err uses (ErrorStreamFilter, reused rather than copied), plus a bare fail / fails word marker that only test runs read (ErrorStreamFilter::for_test_run(); the shared ERROR_PATTERNS and rtk err are untouched) -> [FAIL] DETECTED FAILURES: capped at CAP_LIST with ... +N more
  • B. otherwise [FAIL] Command failed (exit code: N) plus the last ten non-blank lines (same text as rtk err)

A run that exits 0 keeps today's tail, and recognised runners that print a summary are byte-identical (golden tests). The tee recovery hint on failures comes from the existing print_with_hint, and never_worse still applies. A recognised runner that died before its summary (for example a cargo test compile error, exit 101) now shows the error[E0425] block instead of the last 5 lines.

Changes from the first version of this PR, found by re-running #2420's exact script:

  • it did not fix the issue's own repro, because FAIL: isn't in ERROR_PATTERNS; hence the bare-fail marker
  • it ran the new path regardless of exit code, so a green run printing warning: and Summary: 0 failed, 30 passed got a false [FAIL] DETECTED FAILURES: banner
  • the third tier ([ok] All tests passed (no recognized test runner)) is dropped: it claims something rtk doesn't know (cargo test --no-run and pytest --collect-only land there too) and it discards the tail that carries things like 12 tests collected. Restoring it is a three-line change if you prefer it.

The code has moved to src/core/runner.rs since the first version (run_test_cmd / extract_test_summary), so it is re-applied there.

Before/after on #2420's script (rtk test ./vt.sh, exit 1):

develop : OUTPUT (last 5 lines): PASS: case_46 ... PASS: case_50
this PR : [FAIL] DETECTED FAILURES:
            FAIL: test_login broke

Tests

  • 9 unit tests, including the golden table with an exit-code column (expected outputs untouched) and the bare-fail word boundaries (test_fail_case does not match)
  • tests/generic_test_runner_test.rs (unix, 3 e2e tests): rtk test shows only the last 5 output lines (not failures) for unrecognized runners #2420's script, a green run is never reported as failed, and error blocks
  • all new tests fail against the old fallback
  • cargo fmt --all -- --check, cargo clippy --all-targets (0 warnings), cargo test --all (all pass)

For a runner rtk has no parser for (`make test`, a custom script),
`rtk test` printed the last five lines of its output whatever the exit
code, so a failure above a passing tail was hidden: `FAIL: test_login
broke` followed by fifty `PASS:` lines showed only the PASS lines.

When such a run exits non-zero, show what the output says went wrong:

- the error blocks `rtk err` shows (the shared ERROR_PATTERNS, untouched),
  plus a bare "fail" word, which only test runs read;
- with none of those, the exit code and the last ten lines.

A run that exits 0 keeps its tail: "warning" or "0 failed" in a green run
is not a failure. Runners rtk recognises are unchanged.

`run_test_cmd` now goes through `run_filtered_with_exit`, so the summary
sees the exit code.

Fixes rtk-ai#2420
@guyoron1 guyoron1 changed the title fix(test): replace generic runner last-5-lines fallback with 3-tier detection fix(test): surface the failing lines when an unrecognised runner fails Sep 21, 2026
@guyoron1
guyoron1 force-pushed the fix/test-generic-runner-fallback branch from 3c68a32 to fbdc006 Compare September 21, 2026 12:20
@guyoron1

Copy link
Copy Markdown
Contributor Author

Rebased onto current develop. The code moved to src/core/runner.rs (run_test_cmd / extract_test_summary), so I re-applied it there. Re-testing with the exact script from #2420 showed my first version still hid FAIL: test_login broke (ERROR_PATTERNS has no bare FAIL), and it printed a [FAIL] banner for green runs that merely print "warning" or "0 failed".

Now only a non-zero exit takes the new path: the rtk err error blocks plus a bare-fail marker that only test runs read (the shared ERROR_PATTERNS are untouched), otherwise the exit code plus the last ten lines. A run that exits 0 keeps its tail, and recognised runners are unchanged. Unit and e2e tests fail against the old fallback.

Two notes for review: I dropped my third tier ([ok] All tests passed), since it claims something rtk doesn't know and discards the tail that carries things like 12 tests collected; restoring it is a three-line change if you prefer it. And #4125 also edits src/core/runner.rs, so expect a rebase for whichever lands second.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rtk test shows only the last 5 output lines (not failures) for unrecognized runners

1 participant