[pthombre]

What changed and why

At a checkpoint, the process supervisor (#5268) freezes the harness with SIGSTOP and continues it with SIGCONT afterwards. That is needed when the snapshot copies only files, as on OpenSandbox. But a process in uninterruptible sleep cannot be stopped, so the pause is then not confirmed and the harness is killed and relaunched.

A VM memory snapshot (AgentENV, #5410) captures the whole sandbox at one instant, processes included, so the harness does not need to be frozen for a consistent copy. This PR adds a pause that holds the harness without signalling it, and records what a later restore needs to find the paused launch again.

Hold pause

• Hold mode in the process supervisor acknowledges a pause at once and sends no signal. As with a stop pause, the launch's deadline stops counting until the continue arrives. The harness's model calls are held by the model relay instead (feat(agent-utils): resume a supervised harness in place after a restore that kept its processes #5412). The default stays stop.
• SandboxSession(pause_mode=...) launches the supervisor in that mode.

What each acknowledgement records

Every pause and continue acknowledgement now carries:

• mode: stop or hold.
• identity: the supervisor's PID and process start time.
• wall and uptime: the sandbox's wall clock and its seconds since boot when the acknowledgement was written.

SandboxSession.checkpoint_state() records the held pause, its identity, and pause_wall and pause_uptime. After a restore of a process-keeping snapshot, #5412 uses these to recognise the same launch and to set the clock back: AgentENV sets a resumed guest's wall clock to the host's, while uptime continues from the snapshot.

Checkpointer record

The sandbox checkpointer (#5265) records whether the provider's snapshots keep processes (keeps_processes). Older records read as false.

How it works

sequenceDiagram
  participant G as Gym (prepare and resume)
  participant S as Process supervisor
  participant H as Harness (OpenCode)
  G->>S: pause N
  alt stop mode (OpenSandbox)
    S->>H: SIGSTOP
    S-->>G: ack with seq N, mode stop, identity, wall, uptime
  else hold mode (AgentENV)
    Note over H: keeps running unless frozen (PR 5412), model calls held by the relay
    S-->>G: ack with seq N, mode hold, identity, wall, uptime
  end
  Note over G: snapshot the sandbox
  G->>S: continue N
  S->>H: SIGCONT (stop mode only)

Loading

Where this sits in the stack

This stack checkpoints environment-server episodes whose agent harness runs in a sandbox the resources server lends. PRs 1 to 7 add it for OpenCode and SWE-bench on OpenSandbox. PRs 8 to 11 add AgentENV (E2B API, VM memory snapshots), where a restored sandbox still runs the harness's processes, so OpenCode continues in place instead of being relaunched, with the sandbox's clock kept continuous. Each PR's diff shows only that PR's change.

1. feat(checkpoint): protocol for harnesses that borrow a sandbox #5264: feat(checkpoint): protocol for harnesses that borrow a sandbox
2. feat(sandbox): snapshot, fork and delete for OpenSandbox, with a checkpointer #5265: feat(sandbox): snapshot, fork and delete for OpenSandbox, with a checkpointer
3. feat(resources): sandbox owner base for resources servers #5266: feat(resources): sandbox owner base for resources servers
4. feat(swebench): typed sessions on the owner base with capture-once verify #5267: feat(swebench): typed sessions on the owner base with capture-once verify
5. feat(opencode): pause at checkpoints and resume after a crash #5268: feat(opencode): pause at checkpoints and resume after a crash
6. test(checkpoint): end-to-end harness scenarios, SWE-bench recipe, docs #5269: test(checkpoint): end-to-end harness scenarios, SWE-bench recipe, docs
7. feat(swe): repository-task servers on the sandbox owner base #5331: feat(swe): repository-task servers on the sandbox owner base
8. feat(sandbox): AgentENV snapshots, lease and kill-at-timeout for the E2B provider #5410: feat(sandbox): AgentENV snapshots, lease and kill-at-timeout for the E2B provider
9. feat(checkpoint): hold-pause a harness when the sandbox snapshot keeps its processes #5411: feat(checkpoint): hold-pause a harness when the sandbox snapshot keeps its processes (this PR)
10. feat(agent-utils): resume a supervised harness in place after a restore that kept its processes #5412: feat(agent-utils): resume a supervised harness in place after a restore that kept its processes
11. feat(opencode): resume OpenCode in place on sandboxes whose snapshots keep processes #5413: feat(opencode): resume OpenCode in place on sandboxes whose snapshots keep processes

Issue

No tracking issue exists. This is the checkpoint-side half of AgentENV in-place resume. #5412 and #5413 build on the pause mode and the recorded identity, wall clock and uptime.

Validation

• Unit tests for the hold pause (immediate acknowledgement, no signal, deadline credited), the new acknowledgement fields, checkpoint_state(), unchanged stop mode, and the keeps_processes record; existing suites pass; lint is clean.
• Live tests against AgentENV: a held harness kept running through the snapshot, and the continue was acknowledged in the forked sandbox.
• End-to-end checkpoint-and-resume RL training at 8 and 16 nodes: checkpoints committed at scale, sessions resumed in place with a continuous clock, no corrupted token history, and training continued.
• Rebased onto recent main: the stack's suites pass, and two 16-node AgentENV RL training runs of the rebased stack, a checkpoint run and a restore from its checkpoint, commit every checkpoint with no sandbox loss; the restore resumes in place every held session whose image can run the model relay.

Compatibility

• Default unchanged. pause_mode defaults to stop, so OpenSandbox and every existing harness behave as before.
• New acknowledgement fields are additive; readers ignore fields they do not know.
• Checkpoint state gains optional pause_wall and pause_uptime. Checkpointer records gain keeps_processes, which reads as false when absent.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.