[yaoyu-33]

What and why

SWE-bench (#4277), SWE-bench Pro (#3718), SWE-Gym and R2E-Gym (#4302) and Scale-SWE already serve typed resources sessions: their seed starts the task sandbox and lends it through sandbox_access, verify grades the captured patch in a fresh sandbox, and /close_session stops what is left. Their OpenCode configs, though, still reach opencode_sandboxed_agent through the legacy_agent relay, so these benchmarks run with one harness only.

This PR moves the OpenCode configs of these benchmarks onto single_agent_turn and keeps the relay configs under new names:

Config (single_agent_turn + opencode_agent)	Relay config, renamed and deprecated	Taskset
benchmarks/swebench/verified/opencode.yaml	benchmarks/swebench/verified/opencode_legacy_agent.yaml	swebench:verified
benchmarks/swebench/multilingual/opencode.yaml	benchmarks/swebench/multilingual/opencode_legacy_agent.yaml	swebench:multilingual
benchmarks/swebench/pro/opencode.yaml	benchmarks/swebench/pro/opencode_legacy_agent.yaml	swebench:pro
resources_servers/swe_gym/configs/swe_gym_opencode.yaml	resources_servers/swe_gym/configs/swe_gym_opencode_legacy_agent.yaml	swe_gym
resources_servers/r2e_gym/configs/r2e_gym_opencode.yaml	resources_servers/r2e_gym/configs/r2e_gym_opencode_legacy_agent.yaml	r2e_gym
resources_servers/scale_swe/configs/scale_swe_opencode.yaml	resources_servers/scale_swe/configs/scale_swe_opencode_legacy_agent.yaml	scale_swe

The benchmark and environment are already in each path, so the session configs take the plain opencode names. Configs can be referenced and extended elsewhere, so the relay configs are renamed rather than removed; their contents and instance names (<prefix>_opencode_sandboxed_agent, <prefix>_opencode_resources_server) are unchanged apart from a deprecation comment.

Session configs

Each follows verified/hermes.yaml: the datasets move to the resources server with a taskset, the Environment Server binds the resources server and an opencode_agent instance with resources_server: null, resources_tool_transports: [], allowed_agents: [opencode_agent], and sandbox_config.ttl_s: 25200 so the session outlives the 21600-second episode deadline. The benchmark recipes keep three repeats. The training configs turn on the agent's token_id_capture (as opencode_sandboxed_agent does by default) and set the unused golden-patch resources instance to null, as agentic_if does. The opencode_agent README still lists its session path as evaluation only, so the training configs and READMEs ask to check token capture before training on these rollouts.

Renamed relay configs and their citations

• Every citation of the relay recipes now uses the opencode_legacy_agent.yaml paths: benchmarks/nemotron_3.5_super/{eval_container_config,swebench_verified_multilingual}.yaml, its sglang_configs/*.yaml and README, nemotron_recipes/super-3.5/swebench-{verified,multilingual,pro}.yaml, tests/unit_tests/environment/test_benchmarks.py, and responses_api_agents/opencode_agent/tests/test_existing_benchmarks.py. The Nemotron 3.5 Super overlays bind the same instance names, so they need no other change.
• Manifests: opencode/manifest.yaml of each SWE-bench variant moved to opencode_legacy_agent/manifest.yaml with lifecycle: deprecated, and a new opencode/manifest.yaml describes the session recipe (agent_server: opencode_agent).
• Each relay config opens with a deprecation comment that points to its session config; the READMEs of benchmarks/swebench, swe_gym, r2e_gym and scale_swe list the session config first and mark the relay config deprecated. opencode_sandboxed_agent and its legacy_agent /run path are already deprecated ("Integrate Existing Agents").
• SWE-bench Multimodal is left to feat(swebench): run multimodal through environment server sessions #5659.

No issue: this is the config half of the server migrations above.

Compatibility

• benchmarks/swebench/{verified,multilingual,pro}/opencode.yaml and the swebench/*/opencode benchmark names now run opencode_agent sessions. Runs that should reproduce earlier results, such as the Nemotron 3.5 Super recipes, use opencode_legacy_agent.yaml (already switched in this PR). opencode_agent uses different defaults from opencode_sandboxed_agent (thinking on, a 10800-second timeout, subagents enabled), so scores from the two are separate measurements and the session recipes need their own baseline.
• resources_servers/{swe_gym,r2e_gym,scale_swe}/configs/*_opencode.yaml now run opencode_agent sessions; the relay configs are *_opencode_legacy_agent.yaml. The prepare_*.py scripts still stamp rows with the relay agent's agent_ref; collation through the taskset drops it.
• The root README environment table is generated from datasets declared on agents, so it now links SWE-Gym, R2E-Gym and Scale-SWE to their _legacy_agent configs. Listing session configs there needs a change to scripts/update_env_list.py, left as a follow-up.

Validation

• gym env validate --config <file> --model-type vllm_model ...: the six session configs, the three SWE-bench relay recipes, the SWE-Gym and Scale-SWE relay configs, the two Nemotron 3.5 Super overlays and the three nemotron_recipes/super-3.5 SWE-bench recipes report Config is valid. r2e_gym_opencode_legacy_agent.yaml fails as r2e_gym_opencode.yaml already does on main: its dataset license string is not an allowed value (unchanged here).
• gym env validate --manifest <path> for all six SWE-bench manifests, with one-row stand-in dataset files: valid; --sync changed nothing.
• responses_api_agents/opencode_agent/tests/test_existing_benchmarks.py, tests/unit_tests/benchmarks/test_super35_batch_launcher.py, tests/unit_tests/{environment,config,tooling}, tests/unit_tests/cli/test_validate_server_configs.py, tests/unit_tests/data/test_task_data.py, tests/unit_tests/test_fern_docs_links.py: 1995 passed, 17 failed. The same 17 fail on an untouched main checkout on this macOS host (repo tooling and lint tests, nine Super 3.5 launcher tests that hit an unset bash array, and the benchmark listing, which drops 11 existing recipes here; all six SWE-bench recipes are listed).
• pre-commit run --files <changed files>: passed after the README table hook rewrote the table. Commits are DCO signed off.

Rollout evidence

Real-model smokes at head 962d7e8288501874e8a9eb19d5472b1a78814937, one task per config, on a Linux workstation with Docker 28.

• Model: nvidia/nemotron-3.5-super-vl-120b-a12b on the hosted NVIDIA API endpoint through vllm_model, with enable_thinking: false.
• Sandboxes: the Docker provider (nemo_gym/sandbox/providers/docker/configs/docker.yaml) with host networking.
• Overrides: OpenCode timeout: 1200 and max_output_tokens: 16384; clear_swebench_debug_logs: false on the SWE-bench servers. Everything else is the shipped config.
• Commands, per config: gym env start --config run.yaml (the config plus the model and Docker provider configs), then gym eval run --no-serve --config run.yaml --agent <agent> --input <one materialized row> --limit 1 --concurrency 1. The row is the task's prepared row (Verified and Multilingual prepared with their prepare.py; Pro, SWE-Gym, R2E-Gym and Scale-SWE from the server's data/example.jsonl), wrapped with materialize_task(row, taskset=...).
• A watcher listed the nemo-gym.sandbox=1 containers every 5 seconds; containers were listed again after collection, before the servers stopped, and after they stopped.

Config	Task	Reward	evaluation_completed / mask_sample	Patch OpenCode left	Model calls	Sandboxes
swebench/verified/opencode.yaml	django__django-11099	1.0 (resolved)	true / false	901 chars, django/contrib/auth/validators.py	24	task 5–112 s, verifier 117–122 s
swebench/multilingual/opencode.yaml	jqlang__jq-2235	1.0 (resolved)	true / false	440 chars, src/main.c	43	task 5–208 s, verifier 213–268 s
swebench/pro/opencode.yaml	instance_ansible__ansible-a26c325b…	0.0 (not resolved)	true / false	14,900 chars, starting in changelogs/fragments/	58	task 5–189 s, verifier at 194 s
swe_gym_opencode.yaml	iterative__dvc-1651	1.0 (resolved)	true / false	763 chars, dvc/command/status.py	11	task 5–45 s, verifier 51–71 s
r2e_gym_opencode.yaml	pillow__6e97dd5c…	1.0 (resolved)	true / false	411 chars, src/PIL/FontFile.py	5	task 5–45 s, verifier at 51 s
scale_swe_opencode.yaml	more-itertools_more-itertools_pr490	1.0 (resolved)	true / false	2,997 chars, more_itertools/more.py	12	task 5–61 s, verifier at 66 s

In all six runs:

• The servers' access logs show one order: resources /seed_session → /v1/agent_sessions → the agent's /v1/responses → /v1/agent_sessions/close → resources /verify → /close_session → /run returns, every call 200. The agent closed before the patch was captured.
• Two containers of the task image appeared, one after the other, never at the same time: the task sandbox the agent worked in, then a fresh verifier sandbox that graded the captured patch.
• No sandbox container was left after collection, before the servers were stopped, nor after they stopped.
• The Pro result is a measured zero: the patch was graded and did not resolve the task, with evaluation_completed: true and an unmasked reward.

Interrupted run. swe_gym_opencode.yaml on iterative__dvc-1651 again, with NEMO_GYM_MAX_ROLLOUT_ATTEMPTS=1. The watcher killed the task container after OpenCode's 4th model call.

• The agent's /v1/responses answered 500 (OpenCode launch outcome is unknown; cannot confirm termination), and its close answered 500 for the same reason.
• /verify was never called. The resources /close_session returned 200.
• The episode was written to rollouts_failures.jsonl as _ng_failure_class: environment_server_failed, stage agent, not terminal. No reward row was written, so the infrastructure failure was not counted as a zero. gym eval run exited 76.
• No container was left before or after shutdown.

Not covered: training token capture. The hosted endpoint does not return prompt or generation token IDs. The captured model calls hold none, and these runs did not turn on run-level token capture. So the training configs' token_id_capture: true is still unchecked, and inspecting token IDs and log probabilities needs a vLLM endpoint that a borrowed Docker sandbox can reach. The configs and READMEs say so.

Rollouts, server logs, model-call captures and container samples are kept with the run.

Known gaps, left as follow-ups

• scale_swe answers a verify after /close_session with 409, not the masked lost_session_verification result the other SWE servers return.
• r2e_gym_opencode_legacy_agent.yaml (formerly r2e_gym_opencode.yaml) fails gym env validate on its license string, as on main.
• The root README table lists the _legacy_agent training configs (see Compatibility).
• benchmarks/swebench/README.md is also edited by feat(swebench): run multimodal through environment server sessions #5659; whichever merges second resolves a small conflict.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yaoyu-33]

/ok to test 4ef7392

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5762-962d7e828850.docs.buildwithfern.com/nemo/gym

[yaoyu-33]

/ok to test 54f51fd

[yaoyu-33]

/ok to test 1278902

[svcnvidia-nemo-ci]

SHIP WITH CARE

RISK — Provide real-model evidence for the six session recipes

Could you attach representative smoke-rollout evidence for these recipes at the reviewed head? The PR explicitly says the runs haven't happened. This isn't a metadata-only change: the new recipes switch execution to agent sessions, and the training recipes enable token capture. AGENTS.md requires representative real-model smoke rollouts for behavior-changing environment or agent work. Config validation and the reported run on another branch don't establish that all six shipped pairings work. This is a validation gap, not a demonstrated runtime failure.

Correction: Run the Verified, Multilingual, Pro, SWE-Gym, R2E-Gym, and Scale-SWE recipes with a real model and supported sandbox provider. Record the head, commands, task IDs, model/provider settings, rollout artifacts, and inspected results in the PR. No speculative code change is needed unless those runs expose a failure.

Acceptance: For each recipe, show that OpenCode changes the task checkout, agent close finishes before patch capture, verification grades that captured patch in a fresh sandbox, and a completed verification returns evaluation_completed=true and mask_sample=false. Check that both task and verifier sandboxes are released after cleanup. For the three training recipes, enable run-level token capture and inspect captured prompt token IDs, generated token IDs, and corresponding log probabilities; the agent flag alone doesn't establish training readiness. Include a representative interrupted run to check cleanup and ensure an infrastructure failure isn't reported as a completed measured zero.

Preservation: Keep the relay recipes' settings unchanged so existing results remain reproducible. Retain three benchmark repeats and treat session scores as separate measurements. Preserve the existing composition test's assertions: the agent disconnects before verification without destroying the borrowed sandbox, completed results are unmasked, and grader-only fields don't reach the model.

I inspected all 17 changed files and traced the affected config, data-routing, session, verification, and cleanup contracts. I found no additional high-confidence correctness defects. I couldn't execute tests or model rollouts in this read-only review.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

[yaoyu-33]

/ok to test 962d7e8

[yaoyu-33]

Real-model smoke evidence for all six configs at 962d7e8 is now in the PR description (Rollout evidence): each run shows the agent closing before verify, grading in a fresh sandbox, evaluation_completed: true with an unmasked reward (five resolved, one measured zero on Pro), and no sandbox left after cleanup. An interrupted run (task sandbox killed mid-turn) failed as environment_server_failed without reaching /verify, so it was not counted as a zero, and it left no container behind. Not covered yet: token IDs and log probabilities for the three training configs, because the hosted endpoint used here returns no token IDs.