Skip to content

fix(swe_agents): fix exit-code capture and final-failure exit in openhands.sh make-build retry - #2446

Merged
marta-sd merged 2 commits into
NVIDIA-NeMo:mainfrom
michal2409:mfutrega/fix-openhands-make-build-retry
Aug 11, 2026
Merged

marta-sd merged 2 commits into
NVIDIA-NeMo:mainfrom
michal2409:mfutrega/fix-openhands-make-build-retry

Conversation

@michal2409

@michal2409 michal2409 commented Aug 10, 2026 •

Copy link
Copy Markdown
Contributor

The retry loop around make build in setup_scripts/openhands.sh has three bugs:

  1. Wrong exit code captured. exit_code=$? runs after the if statement has completed, so it captures the if-statement's status (always 0), not make's. Timeouts are reported as "failed with exit code 0" and exit 124 is never detected.
  2. Infinite loop on final failure. On the final attempt, a failure neither increments attempt nor exits, so the loop re-runs make build forever instead of failing the setup. In a Docker build this hangs until the CI job times out; worse, if a late retry "succeeds" against a half-broken tree, the failure is deferred to runtime.
  3. Stale .venv between retries. A first-attempt timeout leaves a partially-built in-project .venv behind (POETRY_VIRTUALENVS_IN_PROJECT=true is set just above); make clean does not remove it. The script has already cd'd into $openhands_dir at this point, so the removal only targets the OpenHands checkout.

This PR captures the exit code in an else branch (while it is still make's), exits with it on the final failure, and removes .venv before retrying.

It also makes the first-attempt timeout env-overridable (MAKE_BUILD_TIMEOUT_SECONDS, default unchanged at 2 minutes), so hosts where a healthy make build legitimately takes longer can raise it without editing the script — on such hosts the 2-minute guard otherwise kills every healthy first attempt and forces the fallback path above.

@copy-pr-bot

copy-pr-bot Bot commented Aug 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@michal2409
michal2409 force-pushed the mfutrega/fix-openhands-make-build-retry branch from d3d7c1a to e2c0772 Compare August 10, 2026 18:23
…hands.sh make-build retry

The retry loop around 'make build' in setup_scripts/openhands.sh has three
bugs:

1. 'exit_code=$?' runs after the 'if' statement has completed, so it
   captures the if-statement's status (always 0), not make's. Timeouts are
   therefore reported as 'failed with exit code 0' and never detected as
   exit 124.
2. On the final attempt, a failure neither increments 'attempt' nor exits,
   so the loop re-runs 'make build' forever instead of failing the setup.
3. A first-attempt timeout leaves a partially-built .venv behind;
   'make clean' does not remove it, and the retry can trip over the
   half-installed environment.

Fix: capture the exit code in an 'else' branch (while it is still make's),
'exit' with it on the final failure, and remove .venv before retrying.
Also make the first-attempt timeout env-overridable
(MAKE_BUILD_TIMEOUT_SECONDS, default unchanged at 2 minutes) so hosts
where a healthy build takes longer can raise it without editing the
script.

Signed-off-by: Michal Futrega <mfutrega@nvidia.com>
@michal2409
michal2409 force-pushed the mfutrega/fix-openhands-make-build-retry branch from e2c0772 to 828cee7 Compare August 10, 2026 18:32
@marta-sd
marta-sd enabled auto-merge (squash) August 11, 2026 09:49
@marta-sd
marta-sd merged commit 7c37593 into NVIDIA-NeMo:main Aug 11, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants