[weijiac0619]

What and why

terminal_bench_4/miniswe (Terminal-Bench 4.0) and terminal_bench_science/miniswe (Terminal-Bench-Science v0.1) were the last TB profiles on the legacy_agent /run relay. Each profile now composes three servers, as terminal_bench_science/codex (#5219) does: the TB4 Resources Server, miniswe_sandboxed_agent and single_agent_turn_legacy. The Resources Server owns the task and verifier sandboxes and lends the task sandbox through SandboxAccess. mini-SWE runs in it on an agent session, and the Environment Server closes the agent before grading. The datasets move to the Resources Server instance, and both manifests gain dataset_owner. prompt_source stays agent.

The TB4 Resources Server's session seed lacked three things the legacy seed reply carried. This PR adds them:

• Instruction. Rows carry only the pinned task identity, and the seed returned no instruction. A new instruction_observation setting returns the task's instruction.md as the seed's observation. The Environment Server appends it to the agent's input as one user message, and mini-SWE renders it with its upstream mini.yaml templates. The setting is off by default, so Codex rows, which embed the prompt (prepare_codex.py), are unchanged.
• Task MCP servers and skills directory. A task that declares either gets them in /tmp/.nemo-gym/task.json in its sandbox, the file Harbor episodes write and mini-SWE already reads. In TB 4.0 this covers medical-claims-processing (Playwright MCP) and cumulative-layout-shift (skills). TB-Science has neither.
• Split CPU/GPU endpoints. The seed refused sandbox_split_endpoints. A new sandbox_provider_refs: {cpu, gpu} setting lends the provider block of the pool where the task's sandboxes run. The provider blocks (terminal_bench_4_sandbox{,_cpu,_gpu}, terminal_bench_science_sandbox{,_cpu,_gpu}) move into each benchmark's resources.yaml. The TB-Science Codex block moves there unchanged, so Codex can also run with split endpoints.

miniswe_sandboxed_agent used to log its /run deprecation warning whenever resources_server was set. The migrated profiles set it only to route the dataset, so the warning now fires on the first /run call.

The benchmark settings are unchanged:

• TB 4.0 keeps 500 steps and 30 s per command (Artificial Analysis settings).
• TB-Science keeps Harbor's defaults: no step cap and 30 s per command.
• The agent budget is 8 h. All 136 pinned packages declare 28800 s, and none declares agent.user.

The legacy relay had no episode bound, and single_agent_turn_legacy defaults to 21600 s, so both profiles set one. It covers setup, the agent budget and the slowest verifier: 54000 s for TB 4.0 (fp8-rmsnorm-gemm's verifier allows 18000 s) and 46800 s for TB-Science, the same as its Codex profile.

Relation to #5654. #5654 gives mini_swe_agent agent sessions with an install that meets the installation contract, and leaves the TB4 and TB-Science configs alone. This PR does not depend on it and keeps today's agent, miniswe_sandboxed_agent. The Resources Server changes are agent-independent. #5654 notes that the TB4 Resources Server seed "returns no instruction", which is why its TB4 runs go through the Harbor environment server. With this PR, mini_swe_agent can borrow TB4 sandboxes from the Resources Server too: it reads the task from the input messages and the task context from /tmp/.nemo-gym/task.json. Moving these profiles to mini_swe_agent after #5654 lands is a config change:

• Include responses_api_agents/mini_swe_agent/configs/mini_swe_agent_sandboxed.yaml and _inherit_from: mini_swe_agent.
• Rename harness.step_limit to step_limit and harness.step_timeout_sec to step_timeout.
• Replace agent_timeout_sec / agent_max_timeout_sec with sandbox_runner_timeout_seconds.
• Drop sandbox_model_base_url and artifacts_dir.

The episode bounds already leave room for its 900 s install timeout.

No issue is linked: the work is tracked internally.

Compatibility

• Instance names. terminal_bench_{4,science}_miniswe now names the Environment Server, beside ..._miniswe_agent and ..._miniswe_resources_server. No other config in the repository referenced the old names. gym eval run --benchmark terminal_bench_4/miniswe is unchanged.
• TB4 example config. resources_servers/terminal_bench_4/configs/terminal_bench_4.yaml follows the new names. Its datasets now sit on the Resources Server, so the generated environment table in README.md drops the TB4 row, as it does for other migrated servers.
• Smoke runner. benchmarks/terminal_bench_4/smoke.py posts rows to the profile's single_agent_turn_legacy server.
• New Resources Server settings. Both default to today's behavior (instruction_observation: false, sandbox_provider_refs: {}). Unchanged: Harbor and Codex profiles, and the legacy /run path of miniswe_sandboxed_agent.
• feat(terminal-bench-science): run three more harnesses in the task sandbox #5459 (three more TB-Science harnesses) merges cleanly with this branch. Each of its new profiles defines a terminal_bench_science_sandbox block identical to the one this PR moves into resources.yaml, so the later of the two can drop those copies. refactor(server_runtime): rename the head server to the registry server #5553 (registry-server rename) also merges cleanly, and its smoke.py edit is the only file both touch.

Validation

Unit tests

At aa3fdd5ba, Linux:

• gym env test +entrypoint=resources_servers/terminal_bench_4: 356 passed, 1 skipped.
• gym env test +entrypoint=responses_api_agents/miniswe_sandboxed_agent: 196 passed.
• python -m pytest tests/unit_tests/environment/test_environment_validation.py tests/unit_tests/test_public_api_facades.py tests/unit_tests/test_public_api.py: 944 passed, including the manifest-to-runtime composition of both profiles.

New tests cover:

• The instruction observation when enabled.
• task.json written for a task with MCP servers, and none for a task without.
• The provider block lent for a GPU-pool task, and the 422 when a pool has no block.
• The deprecation warning firing once, on /run only.
• Composition of both profiles: three servers, with lending refs and episode bounds.
• Both TB-Science borrowing profiles (Codex and mini-SWE) sharing the provider blocks.

Real runs

Source: this branch's content before it was committed as aa3fdd5ba (no file changed after the runs). Model: a Nemotron checkpoint served by vLLM through Gym's vllm_model server. Sandboxes: two OpenSandbox deployments, one for CPU and one for GPU, selected per task with ++tb4_split_sandbox_endpoints=true.

Each stage ran one task with gym eval prepare --benchmark terminal_bench_4/miniswe "++prepare_script_args.task_names=[<task>]" and then gym eval run --benchmark terminal_bench_4/miniswe ++tb4_split_sandbox_endpoints=true ++tb4_max_steps=3 ++tb4_agent_max_timeout_sec=900 ++tb4_concurrency=1. The 3-step cap keeps the runs short, so no reward is expected.

Task	Sandbox	Reward	Tests	Health	Model calls	mini-SWE exit	Wall time
ks-solver-cpp	CPU	0	verifier exit 1	healthy	3	LimitsExceeded	463 s
medical-claims-processing (Compose, Playwright MCP)	CPU	0 (partial credit 1.0)	1 of 1	healthy	3	LimitsExceeded	175 s
math-eval-grader	GPU	0	0 of 3	healthy	3	LimitsExceeded	105 s

In every run:

• The model's first input was the official instruction inside mini.yaml's instance template.
• evaluation_completed was true, no sample was masked, and rollouts_failures.jsonl was empty.
• Task and verifier sandboxes ran on the endpoint of the task's pool and were closed, with no cleanup errors.

In the Compose run, the Playwright tool schemas from task.json appeared in the prompt, and all four services were closed. The GPU run's verifier logged 404s collecting /app/extracted.jsonl and /app/grader.py, files the 3-step run never wrote.

Checks

• pre-commit run --files <changed files>: passed.
• DCO sign-off on the commit.

Known limits

• No live TB-Science run. It needs prebuilt agent and verifier images, and upstream publishes none. The profile is covered by config, composition and unit tests, and it uses the same Resources Server path as the TB 4.0 runs above.
• Termination record. TB4's /verify maps termination from response.status, and miniswe_sandboxed_agent sets no status. A LimitsExceeded or timeout exit is therefore recorded as completed in the trial record, as in all three runs above. The actual reason is in the response metadata (miniswe_termination_detail). Reward is unaffected. mini_swe_agent from feat(mini-swe): run mini-SWE in task sandboxes through agent sessions #5654 sets incomplete for these exits, so the switch above would fix this.
• No full benchmark evaluation or score parity with the legacy relay is claimed.

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.