Skip to content

fix(e2e): add --kill-after to run_with_timeout to prevent hung tests - #2982

Closed
gaveezy wants to merge 1 commit into
NVIDIA:mainfrom
gaveezy:fix/nightly-e2e-timeout-no-kill-after-a8dfa27
Closed

gaveezy wants to merge 1 commit into
NVIDIA:mainfrom
gaveezy:fix/nightly-e2e-timeout-no-kill-after-a8dfa27

Conversation

@gaveezy

@gaveezy gaveezy commented May 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The run_with_timeout() helper in test/e2e/e2e-timeout.sh sent only SIGTERM via the timeout command. When a NemoClaw process ignored SIGTERM (e.g., blocked on an uninterruptible SSH child during TC-SBX-08 Process Recovery), the inner per-command timeout was ineffective and the outer overall E2E timeout (2700s) had to clean up, stalling all subsequent test cases with exit code 124.

This adds --kill-after=10 so SIGKILL is sent 10 seconds after SIGTERM, ensuring hung processes are forcibly terminated.

Related Issue

Fixes #2983

Changes

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

  • npx prek run --all-files passes
  • bash -n test/e2e/e2e-timeout.sh passes (syntax check)
  • Tests added or updated for new or changed behavior
  • No secrets, API keys, or credentials committed
  • Docs updated for user-facing behavior changes

AI Disclosure

  • AI-assisted — tool: Claude Code (nemoclaw-diagnosis skill)

Signed-off-by: Hai Nguyen haingu@nvidia.com

The run_with_timeout() helper in e2e-timeout.sh sent only SIGTERM via
the timeout command. When a nemoclaw process ignored SIGTERM (e.g.
blocked on an uninterruptible SSH child), the inner per-command timeout
had no effect and the outer overall E2E timeout (2700s) had to clean up,
stalling all subsequent test cases.

Add --kill-after=10 so SIGKILL is sent 10 seconds after SIGTERM,
ensuring hung processes are forcibly terminated.

Signed-off-by: Hai Nguyen <haingu@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented May 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented May 4, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 04d7b039-f860-4901-819f-2a41d6c32544

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@wscurran

wscurran commented May 4, 2026

Copy link
Copy Markdown
Contributor

✨ Thanks for submitting this PR that proposes a fix to the E2E testing framework by adding a timeout kill after parameter to prevent hung tests. This change improves the reliability of the E2E tests by ensuring that hung processes are forcibly terminated.


Related open PRs:


Related open issues:

@gaveezy gaveezy closed this May 19, 2026
@wscurran wscurran added area: e2e End-to-end tests, nightly failures, or validation infrastructure bug-fix PR fixes a bug or regression and removed E2E labels Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure bug-fix PR fixes a bug or regression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nightly-e2e: sandbox-operations-e2e hangs at TC-SBX-08 due to missing --kill-after in run_with_timeout

2 participants