[gwarmstrong]

What does this PR do?

A create request that times out on the client, or that the caller cancels, can still make its sandbox on the server, and the provider never learns its id. #3807 (now on main) handles this with an attempt-key sweep: it lists the sandboxes carrying the attempt's metadata key and deletes them. The sweep misses a sandbox the server records after the lookup, and finds nothing when the lookup itself fails.

This PR adds @hemildesai's approach under the sweep, with his commit cherry-picked. The create request is not cancelled when the caller gives up. The provider waits on it through asyncio.shield, so the caller still gets its timeout or cancellation on time. Then:

• If the sweep found and deleted the attempt's sandbox, the request is cancelled. The server never answers a create whose Pending sandbox was deleted, so waiting on it would hold the request open until the SDK's request timeout.
• If the sweep found nothing, a tracked task waits for the server's answer and deletes the sandbox it names.

aclose() waits for those tasks up to operations.close_timeout_s, then cancels the rest. The OpenSandbox docs describe both steps.

No issue: found in Terminal-Bench-Science runs, where a 70-task oracle run on OpenSandbox left two sandboxes running until their TTL.

Validation

• tests/unit_tests/test_opensandbox_provider.py: 160 passed after merging main (2026-10-29, 7269de9); 159 before. That includes Hemil's tests and a new one:

  • Hemil's: a create that finishes after the caller gave up (now with a sweep that finds nothing), and aclose() cancelling creates the server never answers.
  • New: a request cancelled once the sweep deleted its sandbox. It fails without the change.

  The other OpenSandbox unit test files pass too, except two --help script tests that need aiohttp on the host Python.

• Live on OpenSandbox at this head: a create timing out after 1 s, a caller cancelling after 1 s, and three attempts that all time out. Each time the sweep found the Pending sandbox and deleted it, aclose() returned at once, and no sandbox was left. When the request was left open instead, the same run held aclose() for the full 120 s request timeout. The found-nothing path doesn't occur on our cell, so only the unit test covers it.

• Re-checked live after merging main (7269de9, 2026-10-29), with the same three cases: a create timing out after 1 s, a caller cancelling after 1 s, and three attempts that all time out. Each Pending sandbox was swept and deleted (five in all), aclose() returned in under 0.03 s, and no sandbox was left.

Rollouts

N/A (provider error handling).

Compatibility and benchmark impact

No settings change. A timed-out or cancelled create whose sandbox the sweep did not find keeps its request open in the background until the server answers, then deletes the sandbox. aclose() waits up to operations.close_timeout_s for those requests.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[hemildesai]

Thanks for tracking this leak down. The diagnosis is right, but I'd like to propose a different fix that needs no create-id label, no list-and-kill sweep, and also covers the case where every attempt fails.

Where the leak actually is

In the SDK (0.1.15), Sandbox.create sets sandbox_id = response.id as soon as the create request returns. From then on, its own except BaseException handler kills the sandbox on any failure, including cancellation. So a sandbox can only leak while the create request is still waiting for its response: our asyncio.wait_for (or a caller's cancel) interrupts it, and the reply carrying the id is thrown away.

Proposal: never cancel the create request

Run Sandbox.create as its own task and wait on it through asyncio.shield. If our timeout fires or the caller cancels, the caller still gets its error on time, but the request keeps running. A tracked background task waits for the response and deletes the sandbox if one comes back:

create = asyncio.ensure_future(Sandbox.create(**kwargs))
try:
    sandbox = await asyncio.wait_for(asyncio.shield(create), timeout=timeout_s)
except BaseException:
    self._reap_abandoned_create(create)  # awaits `create`; on success -> self._cleanup_failed_create_handle(handle)
    raise

_reap_abandoned_create keeps its tasks in a set, and aclose() waits for them before closing the transport. The whole change is about 25 lines in provider.py, and the existing provider unit tests pass unchanged.

Compared with the label approach:

• Exact: it deletes the one sandbox that was abandoned, as soon as the server answers. There are no list calls and no metadata filter.
• Covers every failed attempt: in this PR, the sweep after the retry loop only runs when a later attempt succeeds. If every attempt times out, each attempt's sandbox can appear after that attempt's own sweep listed nothing, and all of them leak.
• Adds no load during overload: the sweep here runs after every failed attempt, including 429s and other clear rejections. Each sweep adds a list request and a new SDK client, which is the most traffic at the worst time.
• No new label on every sandbox, and no extra cleanup holding up a cancelled task.

One requirement: the create request timeout (create.request_timeout_s) must be longer than the server's own create timeout, so the response always arrives. If the connection drops before then, the id is lost, as it is today. Process crashes stay covered by the run attribution label and cleanup_sandboxes.py.

E2E evidence

I tested this against a live OpenSandbox cell (python:3.12-slim, use_server_proxy: true). Create took about 4–8s there, so I set the create timeout to 1s to force timeouts. Each scenario used its own attribution run label; after the create failed, the script waited 90s and listed sandboxes with that label. I ran the same script against unmodified main and against the shield change:

Scenario	main	shield
Normal create, then close	no leak	no leak
Create times out at 1s, no retry	1 still running	no leak
Caller cancels the create after 1s	1 still running	no leak
All 3 retry attempts time out	3 still running	no leak

With the shield, the caller still got OpenSandboxCreateTimeoutError at 1.0s (3.3s across three attempts), or the cancellation at 1.0s. The provider logged Deleting OpenSandbox sandbox … made by an abandoned create for all 5 abandoned creates, about 2–4s after the caller gave up.

Not covered: many creates at once, and connections that drop before the response arrives.

Suggested path

• Replace the create-id label and _delete_create_leftovers with the shield approach. Add a unit test where the create finishes after the caller has given up, and one where the caller cancels it.
• Consider splitting status_failure_window_s (a new opt-in feature) and the deleted-output fix into their own PRs, and link the tracking issue (or say why there isn't one).

Happy to push the prototype branch if it helps.

[gwarmstrong]

Thanks @hemildesai, this is a much better fix. I've switched the PR to your approach:

• Your commit from shield-create-prototype is cherry-picked unchanged (fcf69f3), on top of reverts of the create-id sweep and of the two unrelated changes. The deleted-output fix is now fix(opensandbox): keep a finished command's exit status when it deleted its own output #4871, and the status-failure window is out of this PR. I didn't force-push, so the history shows the reverts; the net diff is your change plus one docs paragraph.
• The docs paragraph (13b2b36) says to keep create.request_timeout_s above the server's create time, the requirement you pointed out.
• I ran your three scenarios live on our cell with this head: the caller got its error after 1.1 s, 1.0 s and 3.0 s, all five abandoned sandboxes were deleted 2 to 4 s later, and none was left.

The description is updated to match.

[gwarmstrong]

/ok to test 13b2b36

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4799-62d8e44de7c0.docs.buildwithfern.com/nemo/gym

[gwarmstrong]

@hemildesai since your review, main picked up #3807, which added an attempt-key sweep to _create_once. It lists the sandboxes carrying the attempt's key, deletes them, and reports one still Pending. I merged main and kept both: the sweep runs first, and your shielded request covers what it misses, which is a sandbox the server records after the lookup, or a failed lookup.

One interaction showed up live. The server never answers a create whose Pending sandbox the sweep deleted, so the open request held aclose() for the full 120 s request timeout. 7ba7057 cancels the request when the sweep found its sandbox and keeps it only when the sweep found none. With that, your three scenarios return at once and leave nothing behind.

The docs paragraph no longer asks for create.request_timeout_s above the create time, because main now caps it at ready_timeout_s on purpose.

[gwarmstrong]

/ok to test 7ba7057

[gwarmstrong]

/ok to test 7269de9

[gwarmstrong]

/ok to test 62d8e44