Skip to content

fix(test): preserve diagnostics from generic runner failures - #3353

Closed
ricorna wants to merge 1 commit into
rtk-ai:developfrom
ricorna:fix/generic-test-failures
Closed

ricorna wants to merge 1 commit into
rtk-ai:developfrom
ricorna:fix/generic-test-failures

Conversation

@ricorna

@ricorna ricorna commented Aug 1, 2026 •

Copy link
Copy Markdown

Summary

  • Pass the child process exit code into generic test summarization and treat it as authoritative.
  • Bypass the shared never-worse raw fallback on nonzero generic-test exits so short success-looking or empty raw output cannot erase the failure banner.
  • Recover failure diagnostics from the complete raw output rather than the displayed tail.
  • Keep bounded head/tail context plus anchor-centered context around detected errors and recognized specialized failures, including diagnostics displaced by cleanup tails.
  • Ignore zero-valued counters such as 0 failed when locating real diagnostics.
  • Keep standalone FAIL: recognition scoped to rtk test so rtk err behavior is unchanged.
  • Add subprocess and unit regressions for early failures, heuristic runner-name collisions, contradictory zero-exit text, success-looking nonzero summaries, zero-failure counters, recognized failures with nearby causal diagnostics, and unknown failure formats.

Safety contract

  • Exit code 0 never receives a failure banner solely because output contains words such as FAILED.
  • A nonzero exit always receives a failure presentation and retains bounded actionable context.
  • Full raw output remains available to RTK tee handling.
  • Never-worse output capping remains active.

Verification

  • cargo fmt --all -- --check
  • cargo clippy --locked --all-targets -- -D warnings
  • cargo test --locked
  • cargo test --locked --test generic_test_runner_test

The complete locked suite passed locally: 2,515 unit tests passed, 8 ignored, plus all integration targets.

Fixes #2420

This overlaps earlier work in #2427, #2683, and #2996, but adds full-output diagnostic recovery, authoritative exit-status handling, and regression coverage for success-looking nonzero output and zero-exit scary words.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@ricorna
ricorna force-pushed the fix/generic-test-failures branch 6 times, most recently from 4c9727a to 6facebd Compare August 1, 2026 07:16
Signed-off-by: Ric Orna <33195092+ricorna@users.noreply.github.com>
@ricorna
ricorna force-pushed the fix/generic-test-failures branch from 6facebd to 6674f9c Compare August 1, 2026 07:24
@KuSh

KuSh commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Closing in favour of #2683, which fixes the same generic runner fallback in extract_test_summary for the same issue and keeps the shared error pattern list untouched. Review and comments are welcome there. If part of the scope here is not covered by #2683, say so on that PR or reply here.

@KuSh KuSh closed this Sep 6, 2026
@KuSh KuSh added the duplicate This issue or pull request already exists label Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

duplicate This issue or pull request already exists

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rtk test shows only the last 5 output lines (not failures) for unrecognized runners

3 participants