Skip to content

fix(opensandbox): retry a timed-out sandbox reconnect - #4109

Open
wprazuch wants to merge 2 commits into
mainfrom
wprazuch/opensandbox-reconnect-retry
Open

wprazuch wants to merge 2 commits into
mainfrom
wprazuch/opensandbox-reconnect-retry

Conversation

@wprazuch

@wprazuch wprazuch commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

What

OpenSandboxProvider.connect now retries when reconnecting to a sandbox times out.

Why

The OpenCode sandboxed agent reconnects to its sandbox by id at the start of a rollout. Under heavy load, with about 1000 sandboxes at once, that call sometimes times out with TimeoutError: [Errno 110] Connect call failed.

  • Sandbox create and commands already retry on errors like this. Reconnect did not.
  • So one timeout turned into a 500 from /run, and the whole eval run stopped. We saw this on SWE-bench Pro on two clusters.

Reconnecting only reads, so trying again is safe.

How

  • It reuses the existing create retry settings (create.retries, retry_delay_s, retry_max_delay_s) and the same retryable-error check.
  • A missing sandbox (404) still fails at once.
  • Each attempt keeps main's health check and cleanup. They moved into _connect_once.

Branch shape

The first commit sits on 1e668906, because evals pinned there need the fix on that lineage. A merge from main follows it, so the PR applies cleanly. The net change to main is provider.py plus the tests.

Tests

  • New: a timeout is retried, retries stop at the limit, and a 404 is not retried. Two of them fail without the fix.
  • test_connect_preserves_failure_when_cleanup_stalls now sets retries: 0, since it tests a single attempt.
  • test_opensandbox_provider.py and test_opensandbox_pty.py: 200 passed.

Under load, reconnecting to a sandbox by id can time out. That call had
no retry, so one timeout failed the rollout with a 500 and stopped the
eval run. Retry it with the existing create retry settings. A missing
sandbox (404) still fails at once.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
Port the reconnect retry onto main's connect, which now also health-checks
and cleans up. The cleanup-stall test pins retries to 0 because it tests
a single attempt.

Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 6, 2026 16:49
@yaoyu-33 yaoyu-33 added area:sandbox Sandbox providers, isolation, and sandbox runtime contracts bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer labels Oct 6, 2026
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Oct 7, 2026

This branch was previously deployed

1 inactive deployment
public — 8b4192aa Deployed Oct 6, 2026 by copy-pr-bot[bot] via release / finalize / notify #4437
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:sandbox Sandbox providers, isolation, and sandbox runtime contracts bug Something isn't working complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants