[pthombre]

What changed and why

On OpenSandbox, a crash restore relaunches OpenCode from its saved conversation (#5268). On AgentENV, whose snapshots keep the sandbox's processes (#5410), the restored sandbox is still running OpenCode mid-task. #5411 and #5412 added the pieces to hold it through a checkpoint and take it over after a restore. This PR wires them into the OpenCode agent, so a restored OpenCode session continues in place: its next model call reaches the new job's model server, and nothing is relaunched.

At launch (only when snapshot_keeps_processes)

• The supervisor runs in hold mode (feat(checkpoint): hold-pause a harness when the sandbox snapshot keeps its processes #5411). With restore_sandbox_clock on (the default), the hold also freezes OpenCode, its tools and the relay at each checkpoint (waiting up to checkpoint_freeze_wait_seconds), so nothing runs on the clock a restore jumps forward.
• Clock sync is disabled at session setup, so a rewound clock stays rewound.
• OpenCode reaches the model through the relay (feat(agent-utils): resume a supervised harness in place after a restore that kept its processes #5412) on sandbox_model_relay_port. If the image has no Python, it uses the direct model URL and that session relaunches on restore.
• The image is checked once for the relay's Python at seed. A session without one logs a single warning naming its task, and records it, so a restore reports relay_unavailable as "no Python in the image".
• The bookmark records keeps_processes, relay_gen and relay_python.

Other providers launch exactly as before.

After a crash (the re-bind)

1. check_restored checks that the restored launch's supervisor is alive with the same identity, still holding the pause, with its relay up.
2. If it can be taken over, the agent sets the clock back (rewind_clock) only when the held pause froze the launch, re-points the relay at the new model URL with a higher generation, continues the pause, and takes the launch over with SandboxSession.adopt. The relay re-sends the cut call, and OpenCode sees only a slow model call. A launch that had already ended is collected the same way.
3. Otherwise, or if the take-over fails, the launch is abandoned (killed, nothing continued) and OpenCode is relaunched from the saved conversation, as on OpenSandbox.

Each outcome is logged so a run can count it. relay_unavailable and relay_gone are expected and logged at INFO; a launch resumed in place, every other reason a launch cannot resume, and the clock decision are logged at WARNING.

Settings

Setting	Default	Notes
sandbox_model_relay_port	fixed port	Sandbox port of the model relay on providers whose snapshots keep processes.
restore_sandbox_clock	true	Freeze at the checkpoint and set the clock back after a restore, keeping it continuous.
checkpoint_freeze_wait_seconds	2.0	How long a freezing pause waits for every process to stop before it stays a plain hold (that restore then leaves the clock).

Also adds a SWE-bench Verified benchmark config for OpenCode sessions on AgentENV, and docs for both restore paths.

How it works

flowchart TD
  CK["checkpoint: hold pause N and freeze,<br/>relay holds model calls,<br/>AgentENV memory snapshot"] --> CR["crash, new job: fork from snapshot,<br/>environment server re-seeds, agent re-binds"]
  CR --> KP{"bookmark keeps processes?"}
  KP -->|"no"| RL["relaunch from saved conversation<br/>(as on OpenSandbox)"]
  KP -->|"yes"| CHK{"check_restored"}
  CHK -->|"RestoredLaunch"| CL["rewind clock if the pause froze"]
  CL --> EP["re-point relay: new model URL,<br/>generation plus 1"]
  EP --> AD["continue pause N, adopt launch"]
  AD --> OK["OpenCode continues in place,<br/>cut call re-sent byte for byte"]
  AD -->|"take-over fails"| AB
  CHK -->|"NotResumable"| AB["abandon launch (KILL, no CONT)"]
  AB --> RL

Loading

Where this sits in the stack

This stack checkpoints environment-server episodes whose agent harness runs in a sandbox the resources server lends. PRs 1 to 7 add it for OpenCode and SWE-bench on OpenSandbox. PRs 8 to 11 add AgentENV (E2B API, VM memory snapshots), where a restored sandbox still runs the harness's processes, so OpenCode continues in place instead of being relaunched, with the sandbox's clock kept continuous. Each PR's diff shows only that PR's change.

1. feat(checkpoint): protocol for harnesses that borrow a sandbox #5264: feat(checkpoint): protocol for harnesses that borrow a sandbox
2. feat(sandbox): snapshot, fork and delete for OpenSandbox, with a checkpointer #5265: feat(sandbox): snapshot, fork and delete for OpenSandbox, with a checkpointer
3. feat(resources): sandbox owner base for resources servers #5266: feat(resources): sandbox owner base for resources servers
4. feat(swebench): typed sessions on the owner base with capture-once verify #5267: feat(swebench): typed sessions on the owner base with capture-once verify
5. feat(opencode): pause at checkpoints and resume after a crash #5268: feat(opencode): pause at checkpoints and resume after a crash
6. test(checkpoint): end-to-end harness scenarios, SWE-bench recipe, docs #5269: test(checkpoint): end-to-end harness scenarios, SWE-bench recipe, docs
7. feat(swe): repository-task servers on the sandbox owner base #5331: feat(swe): repository-task servers on the sandbox owner base
8. feat(sandbox): AgentENV snapshots, lease and kill-at-timeout for the E2B provider #5410: feat(sandbox): AgentENV snapshots, lease and kill-at-timeout for the E2B provider
9. feat(checkpoint): hold-pause a harness when the sandbox snapshot keeps its processes #5411: feat(checkpoint): hold-pause a harness when the sandbox snapshot keeps its processes
10. feat(agent-utils): resume a supervised harness in place after a restore that kept its processes #5412: feat(agent-utils): resume a supervised harness in place after a restore that kept its processes
11. feat(opencode): resume OpenCode in place on sandboxes whose snapshots keep processes #5413: feat(opencode): resume OpenCode in place on sandboxes whose snapshots keep processes (this PR)

Issue

No tracking issue exists. This finishes AgentENV in-place resume for OpenCode, the harness used in SWE training.

Validation

• Unit tests for in-place resume against a new model server, abandon and relaunch, relay versus direct model URL, relay re-pointing, clock rewind only when frozen, and failure paths that must not stop a restore or a seed; existing suites pass; lint is clean.
• Live tests against AgentENV of snapshot, fork and in-place resume with real OpenCode.
• End-to-end checkpoint-and-resume RL training at 8 and 16 nodes: sessions resumed in place with a continuous clock, no corrupted token history, checkpoints committing at scale, and training continuing.
• Rebased onto recent main: the stack's suites pass, and two 16-node AgentENV RL training runs of the rebased stack, a checkpoint run and a restore from its checkpoint, commit every checkpoint with no sandbox loss; the restore resumes in place every held session whose image can run the model relay. Every restored session took the expected path: those with the relay resumed in place, and those whose images have no Python were relaunched from their saved conversation, each named at seed.

Compatibility

• OpenSandbox and other providers whose snapshots do not keep processes: launch, checkpoint and restore are unchanged.
• AgentENV: OpenCode's model calls go through the in-sandbox relay; the task image's time daemons are masked for the session; tools are stopped from the pause to the continue. Set restore_sandbox_clock: false to hold without freezing and leave the clock as AgentENV sets it.
• Images without Python fall back to a relaunch on restore, without losing rollouts.
• Bookmarks gain optional keeps_processes, relay_gen and relay_python. Older bookmarks restore through the relaunch path.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.