[pthombre]

What changed and why

An agent harness such as OpenCode runs a whole rollout as one long command in a sandbox that its resources server lends. This PR makes such an agent checkpointable through a reusable driver, and puts OpenCode on it. At a checkpoint the harness is paused in place, and the same process continues once the checkpoint opens. After a crash, the harness is relaunched in the restored sandbox and continues its saved session.

What this PR adds:

• SandboxedHarnessAgent and HarnessAdapter (nemo_gym/_checkpoint/harness_run.py). The driver owns the checkpointed stage machine of a legacy /run: seed, the harness as one long wait step, verify, and return.

  • Before each step it records a bookmark.
  • Before every launch it asks the owner's /sandbox_access which sandbox is current. After a restore, that is a fork under a new ID.
  • Everything harness-specific belongs to the adapter:
    • where the harness keeps its session (new_state);
    • how to connect to the lent sandbox;
    • whether a saved session can be continued (resume_decision);
    • how to launch;
    • how to verify.

• Pausing a harness at a checkpoint (nemo_gym/_checkpoint/harness.py, optional adapter protocol PausesAtCheckpoints). When a checkpoint asks the run to park (LegacyRun.wait_park_requested(), feat(checkpoint): ask a long-running agent harness to park when a checkpoint opens #4222):

  • the launch is paused: SIGSTOP goes to every process of the launch, and the pause is confirmed once each one is stopped (state T) or dead;
  • the run records a bookmark that continues the harness step and parks;
  • once the checkpoint opens, SIGCONT goes to the same processes, and the run keeps waiting on the same launch.

  Nothing in the sandbox changes while it is paused, so the owner's snapshot (feat(sandbox): sandbox owner base for resources servers, with SWE-bench on it #4576) holds that moment. A held model reply is delivered when the checkpoint opens and read on continue. OpenCode opts in with pause_at_checkpoints: true, the default.

  A pause or continue command gets 120 s. A pause checks for 5 s that every process stopped, and for up to 30 s while the only ones left are in an uninterruptible system call (state D, such as a write to a slow disk), which take the pending SIGSTOP when the call returns. A pause that printed nothing although it ended within its budget is run once more. A pause that fails is logged with the rollout, the exit code, the error, how long it took and the attempt.

• Stopping a harness, the fallback and the path for adapters that do not pause. A stop sends SIGINT, then SIGKILL after 10 s, then SIGKILL again to anything still found; the run relaunches after the resume. The run falls back to the stop when:

  • a pause cannot be confirmed within its checks;
  • a paused launch cannot be continued;
  • the adapter does not pause.

• Finding every process of a launch. A checkpointed launch uploads nemo_gym/agent_utils/process_supervisor.py and, where the sandbox has Python 3.8+ (chosen like the resume helper's interpreter), runs under it.

  • The supervisor gives the harness a process group of its own and adopts every process it orphans (PR_SET_CHILD_SUBREAPER). While it runs, every process of the launch is in its tree: the installer, OpenCode, the commands the model runs in sessions of their own, and background processes they detach.
  • The launch records the supervisor's PID and start time. Pause, continue and stop walk /proc/<pid>/task/*/children from it: one read per process of the launch, never every process of the sandbox. Launches whose commands start thousands of processes made the full scan take minutes, past the command budget (see Rollout evidence).
  • The launch still returns the harness's own exit code, which it reads from the supervisor's receipt.
  • Fallback, without Python 3.8+, when the upload fails or the supervisor cannot launch, and once the supervisor has exited: the launch runs with NEMO_GYM_HARNESS_LAUNCH=<id> in its environment, which every process it starts inherits wherever it is re-parented. A pause or stop then acts on the tagged processes, on those an earlier pass found (by PID and start time), and on all their descendants. A pause of a launch that should be supervised but is not logs a warning naming the rollout.
  • A stop file keeps a launch from starting anything once a stop has beaten it.
  • A pause file makes a launch that has not started yet wait until the checkpoint opens.

• A finished launch keeps its result. OpenSandbox can report a finished execution up to about 30 s late. The launch writes ended into its record when its command returns, so a park request in that window keeps the result instead of pausing or relaunching.

• A cancelled launch is killed, shielded and time-limited, before the cancellation goes on. A paused one is killed too; SIGKILL ends a stopped process.

• OpenCode becomes OpenCodeAdapter. OpenCode keeps its session in SQLite under a per-rollout XDG_DATA_HOME on the sandbox's root filesystem, so the owner's snapshot carries it into a fork.

  • A relaunch re-enters the saved user turn without adding a new one (from fix(checkpoint): wake sandbox command polls while a harness launch is stopped #4553, co-authored by Anish Mahishi). resume_opencode.py runs in the sandbox:

    • it starts an authenticated loopback opencode serve;
    • it posts the session's last user message back with empty parts, which OpenCode 1.17.11 upserts and continues;
    • it checks that the saved user turns are unchanged, then stops the server.

    A checkpoint therefore never adds a message the task did not have.

  • The helper runs on a Python 3.8+ interpreter, chosen from python3, /usr/bin/python3, /opt/miniconda3/bin/python3 and /usr/local/bin/python3; the relaunch fails with a clear message if none exists. A task image's own environment can come first on PATH and be older: SWE-bench's testbed is Python 3.5 or 3.6 for old Django, scikit-learn and astropy, where the helper's from __future__ import annotations is a SyntaxError. OpenCode and the commands the model runs keep the task's environment.

  • A relaunch continues only a session that can be continued. The session must start with the task and hold a completed model response. Otherwise the relaunch starts over with the task in a fresh data folder.

  • _launch_opencode takes the sandbox, the data folder and the resume flag explicitly. responses() keeps working for callers that manage the sandbox themselves, such as the ViBench agent.

• The run is registered before it waits for a concurrency slot. Parked runs keep their slots. A /run queued behind them is now registered at its seed bookmark, so a checkpoint exports it instead of waiting for it. Previously a 32-node run waited out a 1200 s drain on queued runs.

• The owner's sandbox is the owner's. A checkpointed run (checkpointing on, not a restart) never stops the sandbox, because the owner may still export the session.

  • Parked at a bookmark, the run leaves everything to the checkpoint.
  • Verified, the owner has already ended the session.
  • Leaving without a verdict, it closes the session through the owner (POST /close_session with an empty body, feat(sandbox): sandbox owner base for resources servers, with SWE-bench on it #4576).
  • Every other run (a restart, checkpointing off, a lender that cannot capture the session, the agent's own sandbox) stops the sandbox as before.

• Runs that nothing can save are restarts. When the seed reply says the resources server cannot capture the session (x-ng-checkpoint-restart), or no sandbox is lent, the run is marked a restart: it is never parked, and it starts over from its input after a crash.

• A versioned bookmark. The format is {"v": 1, "next": "seed" | "harness" | "verify" | "return", ...}, validated before a continuation runs.

  • An unknown version or a malformed bookmark fails the attempt with an error that names it.
  • The harness bookmark carries the cookies, the verify mode, the sandbox hint, the adapter's state, resume, and a record of every launch.
  • A continuation at verify grades again without launching; one at return returns the stored result.

• A launch log. With checkpointing on, the OpenCode response carries ng_harness_launches: every launch of the rollout, whether it continued a saved session and why, how long it ran, how it ended (finished, interrupted, failed), and how many checkpoints paused it (pauses).

• The second user. The e2e fake_harness_agent (feat(checkpoint): ask a long-running agent harness to park when a checkpoint opens #4222) runs on the driver with a fake adapter, so the process-level harness scenarios exercise the code OpenCode runs on.

At a checkpoint	Paused (default)	Stopped (fallback, or an adapter that does not pause)
The harness processes	SIGSTOP, confirmed stopped	SIGINT, then SIGKILL
The run	records a bookmark and parks while paused	records a bookmark and parks after the launch ended
When the checkpoint opens	SIGCONT: the same process continues	relaunch: resume_decision, then resume_opencode.py
A held model reply	delivered and read on continue	abandoned; re-issued by the relaunch
After a crash	relaunch from the bookmark	relaunch from the bookmark

How it works

Where this PR sits in the overall flow

The highlighted parts are what this PR adds.

flowchart LR
  C["Controller<br/>NeMo RL"]
  CO["Coordination<br/>prepare, commit, restore, resume"]
  subgraph G["One participant per Gym server"]
    A["SandboxedHarnessAgent<br/>bookmark, park, pause or relaunch,<br/>cleanup rule"]
    AD["HarnessAdapter<br/>OpenCodeAdapter"]
    R["Sandbox owner<br/>SWE-bench (#4576)"]
  end
  H["Harness in the lent sandbox<br/>supervised, tagged launch"]
  C --> CO --> G
  A --> AD
  AD -- "launch; pause and continue,<br/>or stop and relaunch" --> H
  A -- "seed, /sandbox_access, verify, close" --> R
  R -- "snapshot at commit, fork at restore" --> H
  classDef this fill:#fde68a,stroke:#b45309,stroke-width:2px,color:#1f2937
  class A,AD this

Loading

The run's stages

stateDiagram-v2
  [*] --> seed
  seed --> harness: lent sandbox, cookies, verify mode<br/>(restart if the owner cannot capture it)
  harness --> harness: park requested: pause, bookmark (resume := true),<br/>continue on open (or stop and relaunch)
  harness --> verify: launch finished (or had ended before the park)
  verify --> return: verify reply recorded
  return --> [*]
  note right of harness
    Before each launch: bookmark (parks here during a checkpoint),
    /sandbox_access, resume_decision when relaunching
  end note

Loading

A checkpoint pauses the launch, and a crash relaunches it

sequenceDiagram
  participant C as Checkpoint
  participant D as Driver
  participant O as Owner (SWE-bench)
  participant H as OpenCode in the sandbox
  D->>D: bookmark {next: harness, resume: false, state: {data_home}}
  D->>O: /sandbox_access
  O-->>D: sandbox S1, workdir
  D->>H: upload process_supervisor.py
  D->>H: env NEMO_GYM_HARNESS_LAUNCH=id bash -c 'python3 process_supervisor.py ... -- bash -c "... opencode run -- task"'
  C->>D: park requested
  D->>H: SIGSTOP every process in the supervisor's tree, until all are stopped
  D->>D: bookmark {resume: true, launches: [interrupted, pauses: 1]} (parks)
  C->>O: commit: snapshot S1 (nothing in it changes while paused)
  C->>D: resume
  D->>H: SIGCONT: the same OpenCode continues, and reads the held model reply
  Note over C,O: later: crash, and the restore forks the snapshot into S2
  D->>O: /sandbox_access (replacement attempt)
  O-->>D: S2
  D->>H: resume_decision: the session starts with the task and has a model response
  D->>H: resume_opencode.py re-enters the saved user turn (no new message)
  H-->>D: finished
  D->>O: /verify (the owner ends the session; the agent never stops S2)

Loading

Leaving the sandbox

Run state when it ends	Checkpointed run	Restart, checkpointing off, lender not migrated, own sandbox
Parked at a bookmark (paused or stopped)	leave everything to the checkpoint	n/a (restarts never park)
Verified	nothing (the owner ended the session)	stop the sandbox
Failed or cancelled before verify	kill the launch (paused or not); POST /close_session {}; release the client	kill the launch; stop the sandbox

Where this sits in the stack

Stack #4578, on ananthsub/partial-ckpt-fault-tests:

1. feat(sandbox): sandbox and snapshot lifecycle for partial-rollout checkpoints #4220: the sandbox session checkpointer, release, the snapshot index, the OpenSandbox lease
2. feat(checkpoint): process-level evidence for resources-owned sandbox checkpoints, their release, and lease expiry #4221: process-level evidence for resources-owned sandboxes
3. feat(checkpoint): ask a long-running agent harness to park when a checkpoint opens #4222: the park request for long-running agent harnesses
4. feat(sandbox): sandbox owner base for resources servers, with SWE-bench on it #4576: the sandbox owner base, with SWE-bench and the notes server on it
5. feat(checkpoint): harness driver for sandboxed agents that pauses them at checkpoints, with OpenCode on it #4274 (this PR): the harness driver with pause-at-checkpoint, with OpenCode on it
6. fix(checkpoint): leave the calls a checkpoint abandoned out of the capture ledger #4451: calls a checkpoint abandoned stay out of the capture ledger
7. fix(checkpoint): wake sandbox command polls while a harness launch is stopped #4553: wake sandbox command polls while a harness launch is stopped; checkpoint-resume docs
8. feat(checkpoint): prepared verification, so a checkpoint never waits for SWE-bench grading #4778: prepared verification, so a checkpoint never waits for SWE-bench grading
9. feat(sandbox): take a failed checkpoint snapshot again before failing the export #4779: retry a failed checkpoint snapshot before failing the export
10. fix(token-id-capture): match a tool call OpenCode rewrote as unavailable #4936: match a tool call OpenCode rewrote as unavailable

Issue

Part of #3284 (sandboxed blackbox agents) and #3269 (OpenCode). Later harnesses (Claude Code through harness_agent, Codex, Pi) implement only an adapter, and opt in to pausing with pauses_at_checkpoints().

Validation

On this head (e832faaf5): this PR now ends with three commits that run the launch under the process supervisor (ecb45e063, b7be6aeb5, e832faaf5). The stack above it was replayed onto them; its top's tree is unchanged from the tested state. In the rl-gym.nightly container on Slurm (job 8281677), on every level of the stack:

• Gym unit tests, OpenCode and harness suites: 1349 passed at this PR, then 1349, 1369, 1372, 1375 and 1379 up to fix(token-id-capture): match a tool call OpenCode rewrote as unavailable #4936; 0 failed at any level.
• ruff check and ruff format --check clean on the changed files at every level.
• test_checkpoint_harness.py now has 21 test functions. The stop scenarios (installing, a command in its own session, a detached process, an untagged child, a harness that ignores SIGINT) run both supervised and unsupervised. New tests: the supervised launch returns the command's output and exit code; a supervised pause stops every process, including a detached daemon and one in another session, and continue resumes them; a launch without an upload runs unsupervised; a launch that cannot run the supervisor still pauses and logs that it is unsupervised; pause and continue get the 120 s budget; an empty pause is retried once; a pause that keeps printing nothing reports why and stops the launch.
• In OpenSandbox sandboxes (job 8267403; swemer_v1 and swe_rebench images, 1 CPU, Python 3.10, kernel 6.12): under the supervisor, 1,000 processes paused in 0.15 s and 2,400 nested ones in 0.24 s, and a detached daemon and a process in another session were stopped too. All resumed on continue, and the supervisor's stop confirmed cleanup. On a compute node the full scan took 26 s to pause 1,000 processes (job 8266603).

Earlier heads (f99ad450e, the fix commit rebased with the stack onto ananthsub/partial-ckpt-fault-tests at 02030c3f8; tests below ran on the pre-rebase head 07070d83f and again on the rebased stack, in the rl-gym.nightly container on Slurm) (f99ad450e, the fix commit rebased with the stack onto ananthsub/partial-ckpt-fault-tests at 02030c3f8; tests below ran on the pre-rebase head 07070d83f and again on the rebased stack, in the rl-gym.nightly container on Slurm):

• OpenCode, SWE-bench, harness and OpenSandbox-polling suites on the whole stack (feat(sandbox): take a failed checkpoint snapshot again before failing the export #4779's head, job 8100572): 186 passed, 5 skipped, 0 failed. This includes test_the_resume_python_selection_skips_a_task_python_older_than_3_8, and the relaunch test checks that the helper is started through the selection.
• The earlier head (c67f810d5, job 8085408) passed 2297 tests across resources_servers/swebench/tests, responses_api_agents/opencode_sandboxed_agent/tests, and every tests/unit_tests file for checkpoint, sandbox, OpenSandbox and token capture. The new commit changes only the relaunch command.
• Every commit carries Signed-off-by.

tests/unit_tests/test_checkpoint_harness.py, 14 test functions with real bash, /proc and signals:

• Stop:
  • during the install, a command in its own session, a detached process, a child that cleared its environment, a harness that ignores SIGINT, and before the launch;
  • other launches are left alone;
  • a launch that ended before the park request keeps its late result;
  • cancelling kills the launch and the detached process it started.
• Pause (new):
  • while paused, every process stays stopped and the files do not change;
  • after two pauses the same launch finishes with no step lost and no relaunch;
  • a pause before the launch starts holds it until the checkpoint opens;
  • a launch that ended before the pause keeps its result and does not park;
  • an unconfirmed pause falls back to the stop;
  • a failing park kills the paused launch;
  • cancelling a paused run kills everything it started.

tests/unit_tests/test_checkpoint_harness_run.py, 20 test functions through the real agent participant:

• the earlier driver tests, and a run queued for a slot that is exported at its seed bookmark and runs after the resume;
• pause (new):
  • a paused launch is exported at its bookmark, continues in place after each of two checkpoints, and is launched once;
  • a checkpoint that closes before the run wakes finds it still paused;
  • the exported bookmark of a paused launch relaunches on the saved session after a crash;
  • an adapter that does not pause gets no park callback.

OpenCode: test_resume_opencode.py and the OpenCode checkpoint tests cover:

• re-entering the saved user turn without a new message;
• starting over when the saved session has no model response.

Live, in OpenSandbox (job 8077144): real OpenCode 1.17.11 on a SWE-bench image, launched as Gym launches it, paused 4 times: mid-tool, waiting on a held reply, 120 s, and mid-stream.

• Files, opencode.db, -wal, -shm and every process's CPU time were unchanged while paused.
• The same OpenCode process finished with no repeated model call.
• A snapshot taken while paused forked byte-identical, and resume_opencode.py continued from it.

Rollout evidence

Pausing with the supervised launch: 16-node full-SWE runs (4 training and 12 generation nodes, 2,048 rollouts in flight on 5 SWE task routes, a checkpoint attempt every 300 s, 2 h each):

Run	Process search	Failed pauses	How they failed	Poisoned model calls
job 8208396 (fresh start)	full /proc scan	25 (19 at one checkpoint)	no output within the 30 s budget (the logs then gave no reason)	212, in 5 rollouts
job 8213273 (resumed)	full /proc scan, 120 s budget	14	killed at the budget (exit -1, stderr 'signal: killed', about 150 s), all on a few tasks	32, in 1 rollout
job 8270615 (fresh start)	supervisor tree walk	3	each reported one process in disk I/O (PENDING <pid>:D) after 6.5–9 s	0

• In job 8270615, 14 checkpoints committed with 0 failures, and 3 training steps completed. Each failed pause fell back to stop and relaunch without poisoning.
• Every poisoned call in the scan-based runs followed a failed pause: the relaunch repeated requests whose parents could not be resolved.
• Job 8270615 ran the first two of this PR's three new commits. The third (e832faaf5) lets a pause wait up to 30 s for processes in disk I/O, the cause of its 3 failures, and logs a launch that falls back to the scan.

32-node GRPO runs: nano-3.5 SWE policy, SWE-bench, 64 prompts × 2, 5 steps.

• The whole stack on top, feat(sandbox): sandbox and snapshot lifecycle for partial-rollout checkpoints #4220 → feat(sandbox): take a failed checkpoint snapshot again before failing the export #4779 (job 8093570; Gym c670de9cc, nothing else):
  • 5/5 steps; 128, 128, 128, 127, 128 valid samples (one sample masked for a logprob error); run_complete=True;
  • 18 checkpoints committed with 0 failures; prepare 9.8 s median, 32 s max;
  • 764 OpenCode sessions and 15,050 tool calls: 0 relaunched sessions, 0 interrupted tools, 0 tools left running;
  • 824 lineage files: 0 dangling parents, 0 branches, 0 token breaks.

Two earlier runs used this PR plus a cut-epoch change in the model gate and checkpoint-timing logs, neither of which is in the stack:

• In-job checkpoints (job 8078235):
  • 5/5 steps, 128/128 valid samples every step, run_complete=True;
  • 19 checkpoints committed with 0 failures; 1,034 snapshots with 0 failures; prepare median 24 s, down from 31 s with stop and relaunch;
  • in 765 OpenCode sessions (15,138 tool calls), 0 tools were interrupted and 0 were left running;
  • all 590 calls that spanned a checkpoint grew by +2 messages, except one that grew by +3 (two parallel tool results). No turn was injected.
• Crash and restore (job 8080199 cancelled after its first rollout snapshot, then 8081456 restored from it): 5/5 steps, run_complete=True. Of the 59 OpenCode runs paused in the snapshot:
  • 37 continued their saved session, attached to their kept calls;
  • 3 that had made no model call started fresh;
  • seed, verify and return bookmarks restored correctly.

Relaunch after a crash restore, fixed in this PR's last commit. Before it, some relaunched OpenCode runs exited about 2 s after starting and made no model call: 18 of 55 in the restore above. All 18 were Python 3.5/3.6 SWE-bench tasks, and all 37 that continued had newer Pythons. The relaunch ran python3 resume_opencode.py, python3 was the task's testbed interpreter, and the helper failed with SyntaxError: future feature annotations is not defined before touching the session. The rollout was then graded as if the model had stopped at the checkpoint.

• Reproduced in OpenSandbox (job 8096675, django__django-10914, testbed Python 3.6.13). With python3 the helper exited 1 after 2.2 s with that SyntaxError and made 0 model requests. With the fix it exited 0 after 28.6 s, made 2 model requests, and the session ran to stop.
• 32-node crash and restore with the fix (job 8099716 cancelled after its first rollout snapshot, then 8099718 restored from it):
  • 5/5 steps; 128, 128, 128, 128, 126 valid samples; run_complete=True;
  • 11 rollout checkpoints with 0 failures;
  • 64 OpenCode relaunches, all exiting 0 and all continuing their session with 3 to 48 new assistant messages (median 13). They include the task versions that failed before: django 1.11 and 3.0-3.2, astropy 1.3, scikit-learn 0.22.

Compatibility

• Unchanged with checkpointing off, and with every lender that cannot capture the session: the run is a restart and stops its sandbox as before.
• With checkpointing on:
  • OpenCode is paused at checkpoints by default. pause_at_checkpoints: false restores stop and relaunch.
  • A checkpointed run no longer stops the owner's sandbox.
  • The response carries ng_harness_launches, whose records now include pauses.
• A checkpointed launch uploads /tmp/nemo-gym-process-supervisor.py and runs under it when the sandbox has Python 3.8+. When the harness exits, the supervisor ends what the harness left running in its own process group. Detached processes in other sessions stay until the sandbox is stopped, as before.
• Paused time counts toward the harness's own timers. OpenCode's 120 s per-command bash timeout can fire right after a long pause. Across 52,027 OpenCode tool calls from earlier runs, 0.02 % ran longer than 60 s.
• The bookmark format is new ("v": 1). The earlier OpenCode boundary keys (next: opencode, sandbox_handle, data_home, continue) are not read; nothing that wrote them has merged.
• New core API:
  • nemo_gym._checkpoint.harness_run: SandboxedHarnessAgent, HarnessAdapter, PausesAtCheckpoints, LaunchContext (with park), LentSandbox, LaunchRecord;
  • nemo_gym._checkpoint.harness, whose run_until_park_requested and LaunchContext take an optional label (the rollout ID) for pause diagnostics.
• OpenCode configuration: it needs output_token_policy: remaining_context, with a context window equal to the served model's max length.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[pthombre]

Rewritten in place (2026-10-09): an explicit snapshot taken at commit is now the only restore point of a checkpointed sandbox. Pause is no longer a restore point: the lazy thaw, resume hooks, restore_point field and listing-based snapshot lookup are gone; a restore always forks the snapshot. A commit the controller stops after carries stop: true and participants park (kill) the exported sandboxes once the write is durable. Episode end deletes the snapshots; a retired attempt keeps them for the retained checkpoints and the sweep reaps the rest. No compatibility with pause-based checkpoint records. Rebased onto the current ananthsub/partial-ckpt-fault-tests. Offline suites green (sandbox 199, checkpoint 608, pilots 69, e2e 65 passed; Slurm job 1167803). Live validation waits on the cell-3 shared snapshot store (fork currently returns SNAPSHOT::NOT_FOUND).

🤖 Generated with Claude Code