feat(data): collate declared tasksets by yaoyu-33 · Pull Request #3818 · NVIDIA-NeMo/Gym

yaoyu-33 · 2026-09-30T16:23:29Z

Expected

A dataset can declare its taskset; normal collation produces environment-neutral tasks without a separate conversion command or importing a particular Environment Server's input model. Split from #3554 per the review request.

Added

Optional dataset taskset and a generic materialize_task shared with single_agent_turn_legacy. It preserves task-owned row fields under task_input, excluding routing/collector metadata.
Each Environment Server validates its own task input. SingleAgentTurnTaskInput accepts both flat fields and the existing {responses_create_params, task_data} shape, without mutating source data; conflicting duplicate fields are rejected.
Preserve task-ID precedence, verifier fields, and collector identity across collation, shards, and retries. Dataset metrics count custom task rows without requiring Responses API fields.
Resolve benchmark discovery, preparation, and manifest validation through the taskset's Environment Server bindings; no legacy agent-to-resources edge is required. Reject conflicting pins/bindings and manifest datasets that resolve to different agents.
Validate source rows before materialization so taskset cannot hide misplaced verifier fields; repeated rows count once in the validation report. Saved-task validation and reverification accept both input shapes.
Regressions for preparation/collation/dispatch, arbitrary custom input, canonical compatibility, repeated rows, source preservation, prompt application, shared metrics sidecars, and reverification. Merging current main preserves its deliverables_dir/reference_ids and shared failure contracts.
Document the two-key YAML configuration, task-ID rules, and unsupported run-level overrides.

Compatibility and benchmark impact

Existing flat-row recipes and source JSONL stay unchanged. This does not switch the Hermes recipe to taskset routing. Newly collated tasksets have flat task-owned fields in task_input; existing nested single-agent inputs remain accepted. For single-agent inputs, task_data remains the canonical container, with non-conflicting flat siblings merged into it.

Collector identity stays outside task_input. Converted rows reject +responses_create_params.* overrides, num_repeats_add_seed, and run-level skills; response parameters must be set before collation. No reward or benchmark-score equivalence is claimed.

Validation

Rebase onto current main (2026-10-05)

Rebased head: 2a09295771a5e1851cf2f76de7ecda37bb22ec5a, based on main 5c6c52b9500883c15a767494842474e7822dcb12.

The rebased tree exactly matches git merge-tree --write-tree 28b3526623e6cbda1c6dcf42fa6c773888693096 5c6c52b9500883c15a767494842474e7822dcb12 (tree 43e19df73422c121dabf0ce0174fb6cc03aa48f7). Existing verifier-context conflict resolutions and their regression tests are preserved.
1,444 tests passed; one local hostname-DNS failure, reproduced on pristine main at the base above: TestDiscoverBenchmarksInDir::test_every_repo_benchmark_appears_in_listing. Both runs discovered 184 benchmark configs but could resolve only 180 because the same four configs hit gaierror. Python 3.13.14 on macOS, reusing dependency-only site-packages with -S to exclude another checkout's editable-install hooks; not a clean dependency bootstrap.
python -m pre_commit run --all-files and git diff --check passed; all nine rebased commits retain their DCO sign-offs.
This is a history-only rebase; no additional runtime change was introduced. The representative real-model smoke noted below remains outstanding. Full repository CI passed on this exact rebased head; see the completed CI evidence below.

Exact regression invocation from the isolated checkout:

PYTHONPATH="$PWD" /Users/yuya/Documents/gym-codex/.venv/bin/python -S -c \
  'import sys; sys.path.append("/Users/yuya/Documents/gym-codex/.venv/lib/python3.13/site-packages"); import pytest; raise SystemExit(pytest.main(sys.argv[1:]))' \
  tests/unit_tests/test_task_materialization.py tests/unit_tests/test_train_data_utils.py \
  tests/unit_tests/test_benchmarks.py tests/unit_tests/test_cli_eval.py \
  tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_global_config.py \
  tests/unit_tests/test_environment_validation.py tests/unit_tests/test_task_data.py \
  tests/unit_tests/test_rollout_reverification.py tests/unit_tests/test_episode_types.py \
  tests/unit_tests/test_environment_server.py environment_servers/single_agent_turn_legacy/tests \
  environment_servers/single_agent_turn/tests \
  --import-mode=importlib -q -o addopts='' -o log_cli=false -p no:cacheprovider --tb=short -rs

Completed CI on the rebased head (2026-10-05)

CICD NeMo Gym run 37386758974 completed successfully on 2a09295771a5e1851cf2f76de7ecda37bb22ec5a.

Core tests: 6,413 passed, 2 skipped, and 226 subtests passed.
Sandbox tests: 913 passed, 5 skipped; combined coverage 95.26%, meeting the 95% gate.
All eight server-suite shards, their aggregate job, and Nemo_CICD_Test passed.
All applicable PR checks completed successfully, including lint, copyright, secrets, DCO, Fern checks/preview, and the build/validate workflow. Container/GPU/provider E2E jobs were conditionally skipped for this non-nightly PR run.
All 15 review threads were checked against the implementation and tests and are resolved. No additional source fix was needed after the rebase.
The representative real-model collation → agent → verifier smoke remains outstanding; CI success does not replace that evidence.

Historical validation before the rebase

Historical source: 8d110a871f7c85b0a7286e07de4d320cd5637652, including main 128eab40a08c5bab71171feb6a55ba294574bd29.

1,371 tests passed on Linux on the exact final source tree. Python 3.13.14, source-bound imports with reused dependency-only site-packages; not a clean dependency bootstrap.
Before/after probe: a custom row without responses_create_params failed with KeyError and counted zero examples at 4ff743fa; the fix preserves the row, counts one example, and leaves the source bytes unchanged.
Before merging main, 1,193 tests passed on macOS. On the final merged tree, macOS passed 1,370 tests with one hostname-DNS failure in the repository benchmark-listing test; the same failure reproduced on pristine main. That test passes on Linux. The first Linux archive run lacked the Git index required by the committed-data sweep; restoring the exact tracked-path index fixed the test setup without source changes, and the full suite above was rerun.
pre-commit run --all-files and git diff --check passed on the final tree. Fern: 0 errors, 1 warning (authenticated redirect checks unavailable). Both new commits are DCO-signed.
Representative real-model collation → agent → verifier smoke remains outstanding. No model endpoint/config was available in this direct fix session; unit/integration checks are not rollout evidence. Full repository CI is also not claimed, and no trust-gate comment, approval, merge, or thread resolution was performed.

python -m pytest tests/unit_tests/test_single_agent_task.py \
  tests/unit_tests/test_train_data_utils.py tests/unit_tests/test_benchmarks.py \
  tests/unit_tests/test_cli_eval.py tests/unit_tests/test_rollout_collection.py \
  tests/unit_tests/test_global_config.py tests/unit_tests/test_environment_validation.py \
  tests/unit_tests/test_task_data.py tests/unit_tests/test_rollout_reverification.py \
  tests/unit_tests/test_episode_types.py tests/unit_tests/test_environment_server.py \
  environment_servers/single_agent_turn_legacy/tests environment_servers/single_agent_turn/tests \
  --import-mode=importlib -q -o addopts='' -o log_cli=false -p no:cacheprovider --tb=short -rs
python -m pre_commit run --all-files
git diff --check
cd fern && npm run check -- --warnings

copy-pr-bot · 2026-09-30T16:23:33Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

yaoyu-33 · 2026-10-02T01:00:08Z

/ok to test 74f16eb

github-actions · 2026-10-02T01:02:18Z

🌿 Preview your docs: https://nvidia-preview-pr-3818-2a09295771a5.docs.buildwithfern.com/nemo/gym

## Expected Harden Hermes sandbox sessions. Data/CLI changes are separate in NVIDIA-NeMo#3818. Implements [MB-1694](https://linear.app/nvidia/issue/MB-1694/runbook-onboard-sandboxed-harnesses-to-native-environmentserver). ## Added - Shared base-agent seed/close retry handling, immutable seed binding, cookie checks, session locks, and close receipts. - Confirmed sandbox process cleanup, launch fencing, internal deadlines, and owned-sandbox teardown. Close confirms cleanup before cancelling borrowed-sandbox execution. - Wall-time stops checkpoint partial results; stale PID signaling is guarded by the remote receipt. Failed setup retains cleanup-only state through a shared API. - SWE-Pro keeps its existing scoring and completion rules. Patch-extraction exceptions remain visible in the task's `error` field without masking the sample. - Identical activation retries join or replay the original run. Provider failures remain retryable and skip verification. - One request validator and prompt/settings builder for host and sandbox execution. ## Concerns - Sandbox prompt composition is unchanged: config system prompt, request instructions, then input system message. Host execution now combines these prompts instead of letting config replace the dataset prompt, and honors request temperature, including zero. `developer` messages now return 422. These host-path changes can change benchmark scores. - Unsupported non-default controls return 422, including `top_p=1.0`, `store`, `service_tier`, and metadata overrides. `model` must match the configured model. `max_output_tokens` is rejected because Hermes does not enforce a total-response budget; config `max_tokens` remains per model call. These 422s are permanent episode errors. - One activation per session, one agent worker, Linux exec-capable sandbox. Close retries must finish within the retention window. Final usage aggregation remains TODO. - Separate follow-up found during smoke: the iteration-limit summary sends `function.call_id` / `function.response_item_id` in tool history, which Gym rejects with 422. The run returns `incomplete`; its existing patch remains gradable. This update does not fix that summary path. - Simple Agent (NVIDIA-NeMo#3784) is not modified here. The SWE-Pro recipe still uses prepared flat rows through `single_agent_turn_legacy` and the EnvironmentServer session lifecycle. ## Validation Latest review follow-up: `e378cd45c` (includes the merge-queue assertion fix in `4df61bf18`). - Removed the redundant session-lock lookup guard and unused thinking-setting diagnostics. Close-requested stops now report `cancelled`, not `wall_time`; deadline behavior and scoring are unchanged. - Five targeted regression cases failed on the previous code. Final focused suite: **563 passed on Linux**; **548 passed, 15 platform skips on macOS**, Python 3.13.14. - `pre-commit run --all-files`, `git diff --check`, and **18 Fern link tests** passed. `cd fern && npm run check`: **0 errors, 1 warning**. Cleanup docs now identify close ordering, not `setsid()` alone, as the guarantee. - Real-model Hermes × SWE-Pro smoke with `nvidia/nemotron-3-super-120b-a12b` and a 15-second runner limit: 3 model calls, `incomplete` / `stop_reason=wall_time`, reward **0**, `mask_sample=false`, `evaluation_completed=true`, 18 verifier tests. Agent close preceded verification; no owned containers/network remained. Runtime file hashes matched the local revision. This is a lifecycle smoke, not an accuracy baseline. <details> <summary>Latest focused test command</summary> ```bash python -m pytest \ environment_servers/single_agent_turn/tests environment_servers/single_agent_turn_legacy/tests \ environment_servers/legacy_agent/tests tests/unit_tests/test_episode_types.py \ tests/unit_tests/test_environment_server.py tests/unit_tests/test_rollout_outcomes.py \ tests/unit_tests/test_failure_kinds.py responses_api_agents/hermes_agent/tests \ responses_api_agents/hermes_sandboxed_agent/tests resources_servers/swebench_pro/tests \ tests/unit_tests/test_agent_session_retries.py tests/unit_tests/test_agent_sessions.py \ tests/unit_tests/test_base_responses_api_agent.py tests/unit_tests/test_process_supervisor.py \ tests/unit_tests/test_local_provider.py \ --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short -rs ``` </details> Scoring revert: `f06ef8dba888fcffaf86eae265fb63f3411db5f6`. - `python -m pytest resources_servers/swebench_pro/tests environment_servers/single_agent_turn/tests environment_servers/single_agent_turn_legacy/tests --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short`: **135 passed**. Four failure-policy regression cases failed before the revert, then passed; extraction errors retain the original exception message. - `pre-commit run --all-files` and `git diff --check` passed. - Real-model SWE-Pro HTTP smoke with `nvidia/nemotron-3-super-120b-a12b`, a 15-second agent limit, and an invalid Git base injected only in the smoke driver: reward **0**, `mask_sample=false`, `evaluation_completed=true`, and the actual extraction exception in `error`. The agent closed before verification; all owned containers/network were cleaned up. This checks failure reporting, not benchmark accuracy. - No documentation change needed: this restores the existing verifier contract rather than adding an API. Earlier review-fix validation at `95a2b6481fa0225fc5a363443d9a536e98cd304f`: - macOS/Python 3.13.14: **494 passed, 13 platform skips**. - Linux/Python 3.13.14: **422 passed** across Hermes, shared sessions, process supervision, SWE-Pro, and Environment Server tests. Final verifier tests: **49 passed**; final supervisor/runner recheck: **40 passed**. - Real-process regressions cover provider-group cancellation, lost spawn handles, detached children, timeout checkpoints (including an uncooperative worker), stale-PID protection, partial setup, and cleanup retries. - Two real-model SWE-Pro HTTP smokes with `nvidia/nemotron-3-super-120b-a12b`: both reached verification after agent close, returned valid reward 0, and left no owned containers or networks. One stopped on model-output truncation; the forced 15-second wall-time run returned `incomplete` / `stop_reason=wall_time`. Both had no patch; these are lifecycle checks, **not accuracy baselines**. Partial-work preservation is separately covered by deterministic subprocess tests. - `pre-commit run --all-files` and diff checks passed; Fern: 0 errors, 1 warning. The test import conflict with main was removed without rebasing. - Full repository CI is not claimed. The inherited DCO mismatch in `07ca331ca1e12d234f1a3a60bb12e1651e496786` remains; the new commit is signed off. <details> <summary>Current focused command</summary> ```bash python -m pytest responses_api_agents/hermes_agent/tests \ responses_api_agents/hermes_sandboxed_agent/tests \ tests/unit_tests/test_process_supervisor.py tests/unit_tests/test_local_provider.py \ tests/unit_tests/test_sandbox_stop_retry.py tests/unit_tests/test_base_responses_api_agent.py \ tests/unit_tests/test_agent_sessions.py tests/unit_tests/test_agent_session_retries.py \ tests/unit_tests/test_environment_server.py resources_servers/swebench_pro/tests \ environment_servers/single_agent_turn/tests \ --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short python -m pre_commit run --all-files cd fern && npm run check ``` </details> <details> <summary>Previous validation at d5f13f5 (historical)</summary> Head: `d5f13f54cf762c7e687dd2d50d2583efe2c9b717`. - macOS/Python 3.13.14: **487 passed, 8 skipped** (Linux-only checks). - Linux/Python 3.13.14: **292 passed**, including Hermes, sessions, process cleanup, and EnvironmentServer tests. Uploaded source and test hashes match this commit. - New regressions failed against the old code: ignored controls and host/sandbox prompt/settings drift. Also test matching model names, zero temperature, unchanged input, validation once per request, and rejection of future non-default schema controls. - `pre-commit run --all-files`, diff checks, and Fern passed (0 errors, 1 warning). New commit is DCO-signed. - Fresh real-model public SWE-Pro HTTP smoke using `nvidia/nemotron-3-super-120b-a12b`: **reward 1.0**, **18/18 verifier tests passed**, 11,273-byte patch. Verified prompt ordering and temperature at the Model Server, agent close before verification, Resources close, and no remaining owned containers/network. Inspected the patch and verifier results. Agent reached the 20-turn smoke limit and returned `incomplete`, with the separate summary-call 422 described above. - This is one functional smoke, not an accuracy/baseline comparison. Reused isolated dependency environments; full repository CI is not claimed. The inherited DCO author/sign-off mismatch in `07ca331ca1e12d234f1a3a60bb12e1651e496786` is unchanged. <details> <summary>Focused commands</summary> ```bash python -m pytest responses_api_agents/hermes_agent/tests \ tests/unit_tests/test_agent_session_retries.py \ tests/unit_tests/test_agent_sessions.py tests/unit_tests/test_base_responses_api_agent.py \ tests/unit_tests/test_process_supervisor.py tests/unit_tests/test_local_provider.py \ responses_api_agents/hermes_sandboxed_agent/tests \ environment_servers/single_agent_turn/tests environment_servers/single_agent_turn_legacy/tests \ resources_servers/swebench_pro/tests tests/unit_tests/test_resources_sessions.py \ tests/unit_tests/test_environment_server.py \ --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short -rs # Linux subset: python -m pytest tests/unit_tests/test_process_supervisor.py tests/unit_tests/test_local_provider.py \ tests/unit_tests/test_agent_session_retries.py responses_api_agents/hermes_agent/tests \ environment_servers/single_agent_turn/tests \ --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short -rs python -m pre_commit run --all-files cd fern && npm run check ``` </details> </details> --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com> Signed-off-by: Adil Asif <adasif@nvidia.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-authored-by: Ananth Subramaniam <ansubramania@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <martas@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Adil Asif <adasif@nvidia.com>

ananthsub · 2026-10-05T13:31:01Z

/ok to test 8d110a8

github-actions · 2026-10-06T02:28:06Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/37404261284

…ent servers NVIDIA-NeMo#3818's resolve_dataset_agent read only the routed environment server's agent_server, so a taskset routed to a server that fronts several agents, such as a user and an assistant, failed collation and benchmark discovery. Resolve through environment_server_agent_refs instead: a single fronted agent resolves as before, a dataset pin chooses among several, and several without a pin is an error that names them. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>

## Expected A dataset can declare its taskset; normal collation produces environment-neutral tasks without a separate conversion command or importing a particular Environment Server's input model. Split from NVIDIA-NeMo#3554 per the [review request](NVIDIA-NeMo#3554 (comment)). ## Added - Optional dataset `taskset` and a generic `materialize_task` shared with `single_agent_turn_legacy`. It preserves task-owned row fields under `task_input`, excluding routing/collector metadata. - Each Environment Server validates its own task input. `SingleAgentTurnTaskInput` accepts both flat fields and the existing `{responses_create_params, task_data}` shape, without mutating source data; conflicting duplicate fields are rejected. - Preserve task-ID precedence, verifier fields, and collector identity across collation, shards, and retries. Dataset metrics count custom task rows without requiring Responses API fields. - Resolve benchmark discovery, preparation, and manifest validation through the taskset's Environment Server bindings; no legacy agent-to-resources edge is required. Reject conflicting pins/bindings and manifest datasets that resolve to different agents. - Validate source rows before materialization so `taskset` cannot hide misplaced verifier fields; repeated rows count once in the validation report. Saved-task validation and reverification accept both input shapes. - Regressions for preparation/collation/dispatch, arbitrary custom input, canonical compatibility, repeated rows, source preservation, prompt application, shared metrics sidecars, and reverification. Merging current main preserves its `deliverables_dir`/`reference_ids` and shared failure contracts. - Document the two-key YAML configuration, task-ID rules, and unsupported run-level overrides. ## Compatibility and benchmark impact Existing flat-row recipes and source JSONL stay unchanged. This does not switch the Hermes recipe to taskset routing. Newly collated tasksets have flat task-owned fields in `task_input`; existing nested single-agent inputs remain accepted. For single-agent inputs, `task_data` remains the canonical container, with non-conflicting flat siblings merged into it. Collector identity stays outside `task_input`. Converted rows reject `+responses_create_params.*` overrides, `num_repeats_add_seed`, and run-level `skills`; response parameters must be set before collation. No reward or benchmark-score equivalence is claimed. ## Validation ### Rebase onto current main (2026-10-05) Rebased head: `2a09295771a5e1851cf2f76de7ecda37bb22ec5a`, based on main `5c6c52b9500883c15a767494842474e7822dcb12`. - The rebased tree exactly matches `git merge-tree --write-tree 28b3526 5c6c52b` (tree `43e19df73422c121dabf0ce0174fb6cc03aa48f7`). Existing verifier-context conflict resolutions and their regression tests are preserved. - **1,444 tests passed; one local hostname-DNS failure**, reproduced on pristine main at the base above: `TestDiscoverBenchmarksInDir::test_every_repo_benchmark_appears_in_listing`. Both runs discovered 184 benchmark configs but could resolve only 180 because the same four configs hit `gaierror`. Python 3.13.14 on macOS, reusing dependency-only site-packages with `-S` to exclude another checkout's editable-install hooks; not a clean dependency bootstrap. - `python -m pre_commit run --all-files` and `git diff --check` passed; all nine rebased commits retain their DCO sign-offs. - This is a history-only rebase; no additional runtime change was introduced. The representative real-model smoke noted below remains outstanding. Full repository CI passed on this exact rebased head; see the completed CI evidence below. Exact regression invocation from the isolated checkout: ```bash PYTHONPATH="$PWD" /Users/yuya/Documents/gym-codex/.venv/bin/python -S -c \ 'import sys; sys.path.append("/Users/yuya/Documents/gym-codex/.venv/lib/python3.13/site-packages"); import pytest; raise SystemExit(pytest.main(sys.argv[1:]))' \ tests/unit_tests/test_task_materialization.py tests/unit_tests/test_train_data_utils.py \ tests/unit_tests/test_benchmarks.py tests/unit_tests/test_cli_eval.py \ tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_global_config.py \ tests/unit_tests/test_environment_validation.py tests/unit_tests/test_task_data.py \ tests/unit_tests/test_rollout_reverification.py tests/unit_tests/test_episode_types.py \ tests/unit_tests/test_environment_server.py environment_servers/single_agent_turn_legacy/tests \ environment_servers/single_agent_turn/tests \ --import-mode=importlib -q -o addopts='' -o log_cli=false -p no:cacheprovider --tb=short -rs ``` ### Completed CI on the rebased head (2026-10-05) [CICD NeMo Gym run 37386758974](https://github.com/NVIDIA-NeMo/Gym/actions/runs/37386758974) completed successfully on `2a09295771a5e1851cf2f76de7ecda37bb22ec5a`. - Core tests: 6,413 passed, 2 skipped, and 226 subtests passed. - Sandbox tests: 913 passed, 5 skipped; combined coverage 95.26%, meeting the 95% gate. - All eight server-suite shards, their aggregate job, and `Nemo_CICD_Test` passed. - All applicable PR checks completed successfully, including lint, copyright, secrets, DCO, Fern checks/preview, and the build/validate workflow. Container/GPU/provider E2E jobs were conditionally skipped for this non-nightly PR run. - All 15 review threads were checked against the implementation and tests and are resolved. No additional source fix was needed after the rebase. - The representative real-model collation → agent → verifier smoke remains outstanding; CI success does not replace that evidence. ### Historical validation before the rebase Historical source: `8d110a871f7c85b0a7286e07de4d320cd5637652`, including main `128eab40a08c5bab71171feb6a55ba294574bd29`. - **1,371 tests passed on Linux** on the exact final source tree. Python 3.13.14, source-bound imports with reused dependency-only site-packages; not a clean dependency bootstrap. - Before/after probe: a custom row without `responses_create_params` failed with `KeyError` and counted zero examples at `4ff743fa`; the fix preserves the row, counts one example, and leaves the source bytes unchanged. - Before merging main, 1,193 tests passed on macOS. On the final merged tree, macOS passed 1,370 tests with one hostname-DNS failure in the repository benchmark-listing test; the same failure reproduced on pristine main. That test passes on Linux. The first Linux archive run lacked the Git index required by the committed-data sweep; restoring the exact tracked-path index fixed the test setup without source changes, and the full suite above was rerun. - `pre-commit run --all-files` and `git diff --check` passed on the final tree. Fern: 0 errors, 1 warning (authenticated redirect checks unavailable). Both new commits are DCO-signed. - **Representative real-model collation → agent → verifier smoke remains outstanding.** No model endpoint/config was available in this direct fix session; unit/integration checks are not rollout evidence. Full repository CI is also not claimed, and no trust-gate comment, approval, merge, or thread resolution was performed. ```bash python -m pytest tests/unit_tests/test_single_agent_task.py \ tests/unit_tests/test_train_data_utils.py tests/unit_tests/test_benchmarks.py \ tests/unit_tests/test_cli_eval.py tests/unit_tests/test_rollout_collection.py \ tests/unit_tests/test_global_config.py tests/unit_tests/test_environment_validation.py \ tests/unit_tests/test_task_data.py tests/unit_tests/test_rollout_reverification.py \ tests/unit_tests/test_episode_types.py tests/unit_tests/test_environment_server.py \ environment_servers/single_agent_turn_legacy/tests environment_servers/single_agent_turn/tests \ --import-mode=importlib -q -o addopts='' -o log_cli=false -p no:cacheprovider --tb=short -rs python -m pre_commit run --all-files git diff --check cd fern && npm run check -- --warnings ``` --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> (cherry picked from commit 325fb49) Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>

…VIDIA-NeMo#4053) ## What does this PR do? Makes Gym and its migration tooling agree on which agents an environment server fronts, including servers that front several agents. It combines the runtime change from NVIDIA-NeMo#4059 with fixes to `scripts/add_legacy_agent_environment_servers.py`, the `ServerRefNotFoundError` rename hint, and the troubleshooting docs, so the environment server and agent server changes land together. The same commits are in the `r0.7.0` cherry-pick of NVIDIA-NeMo#3929 (NVIDIA-NeMo#4050). ## Where this fits Datasets reach an environment server in one of two ways today: run request rows route by their agent, and datasets that declare `taskset:` are materialized during collation and route by `environment_server_routes`. The goal is one path, where every dataset declares a taskset and routes to an environment server: ```mermaid flowchart LR P["Preparation script task rows with a stable task_id"] --> C["Collation validates, computes metrics, materializes"] Y["Dataset YAML file + taskset"] --> C S["task_data.py schemas environment server + resources server"] -.-> C C --> R["Rollout collection routes by taskset"] T["environment_server_routes"] --> R R --> E["Environment server single-agent, multi-agent, or legacy_agent relay"] ``` This PR is one step on the way there. Each step is additive, and datasets without `taskset:` keep working until the last one: ```mermaid flowchart TB S3818["PR 3818 (merged) collation materializes declared tasksets"] S4053["PR 4053 tasksets route to servers with several agents"] S4096["PR 4096 stable, unique task IDs"] S4097["PR 4097 metrics for every row"] S4098["PR 4098 schema per environment server"] S4099["PR 4099 legacy_agent relays serve tasksets"] SNEXT["Later configs declare taskset:, then agent routing retires"] S3818 --> S4053 --> S4096 --> S4097 --> S4098 --> S4099 --> SNEXT classDef done fill:#F1EFE8,stroke:#5F5E5A,color:#444441 classDef this fill:#EEEDFE,stroke:#534AB7,color:#26215C,stroke-width:2px classDef next fill:#E1F5EE,stroke:#0F6E56,color:#085041 class S3818 done class S4053 this class S4096,S4097,S4098,S4099,SNEXT next ``` Gray: merged. Purple: this PR. Green: other steps. ### Environment servers that front several agents An environment server can front several agents, such as the user and assistant agents of the user-simulation server in NVIDIA-NeMo#3800. It references them through fields such as `user_agent` and `assistant_agent`. Gym only recognized the `agent_server` field. For such a server: - The deprecation check from NVIDIA-NeMo#3929 treated its agents as unfronted and generated a `legacy_agent` relay in front of each. - Agent-routed rows could not find the server, because `_environment_servers_by_agent` only read `agent_server`. The first commit from NVIDIA-NeMo#4059 (authored by Felipe Vieira Frujeri) adds `environment_server_agent_refs` to `nemo_gym/global_config.py`. Any top-level field whose value has `type: responses_api_agents` references an agent, and so does `agent_server`, whose type may be left out. The deprecation check, agent routing, and `_retarget_environment_servers` (which follows an agent swapped during composition) use it. Two places in `nemo_gym/rollout_collection.py` still read only `agent_server`: - `_agent_name_for_row` names the agent a row's results are attributed to. It decides whether token IDs are captured and labels aggregate metrics. Results are now attributed to the server's `agent_server`, or to the only agent it references. A server that fronts several agents without an `agent_server` has no single agent to attribute, as before. - `_validate_environment_servers` rejects a row whose `agent_ref` names an agent the server does not front. It skipped the check for any server without `agent_server`. A row may now name any agent its server references. ### The migration script The script assumed an environment server references one agent, through `agent_server`. It now finds agent references with the same rule as `environment_server_agent_refs`. The script depends only on PyYAML, so it keeps its own copy of the rule, and a test checks that both agree. Each referenced agent counts as fronted, so the script no longer adds relays in front of agents a multi-agent server already fronts, such as those in `resources_servers/usersim/configs/usersim.yaml` from NVIDIA-NeMo#3800. For an agent renamed with `_inherit_from`, the renamed agent's server used to inherit from the name the script would generate for the source, always as a `legacy_agent` relay. That failed in three ways: - The source's server has a custom name: the inherited name does not exist, raising `InheritPathNotFoundError`. - The source's server is `single_agent_turn_legacy`: the inherited server gains a second `legacy_agent` block without an entrypoint, raising `AlmostServerError`. - Inheriting retires the source server's name, which `environment_server_routes` may still use. For example, usersim routes its tasksets to `usersim_environment`. The script now handles a rename this way: - Where a declared server references the source, the script points that reference at the new name, on the existing server. The server keeps its name, so routes and the server's other agents are unaffected. Renaming several agents of one server writes one block for that server. - Where the source has only the server the script generates, the renamed agent's server inherits it, as before. - A rename in the same file as a server that references its source cannot be migrated by appending, because the server's key would repeat. The script reports the field to update by hand and exits with 2. ### The renamed-agent hint When an environment server references an agent that no longer exists, `ServerRefNotFoundError` suggests the agent may have been renamed with `_inherit_from`. The hint said the environment server "must be renamed with it", which no longer matches the script. - The hint now says the environment server must reference the agent's new name, and names the field to update instead of always `agent_server.name`. - It is shown for any agent reference on an environment server, not only `agent_server`. - `fern/versions/latest/pages/troubleshooting/configuration.mdx` describes the same rename behavior, and the script's exit code 2 for a rename that needs a manual update. ### Declared tasksets routed to a multi-agent server NVIDIA-NeMo#3818 added `resolve_dataset_agent`'s taskset path, used by benchmark discovery, `gym eval prepare`, manifest validation, and collation. It read only the routed environment server's `agent_server`, so a taskset routed to a server that fronts several agents failed with "must bind a valid agent_server". A taskset routes to an environment server, not to an agent: rollout collection sends materialized rows to the server that `environment_server_routes` names and never uses an agent. So resolution now checks the route instead of choosing an agent: - Every agent the server references, through `environment_server_agent_refs`, must exist. - A dataset declared on an agent must be one of those agents. - A dataset declared on a resources server must be the one the server binds. - A dataset with `taskset:` cannot set `agent:`. Only benchmark datasets accept `agent:`, so requiring a pin would have left example and train datasets for a server with several agents unable to collate, and would have made an environment server that owns its dataset name one of its own agents. It returns the declaring agent, or the server's attributed agent (its `agent_server`, or its only agent), or none when the server fronts several agents without an `agent_server`. `environment_server_agent_names` and `environment_server_attributed_agent` move to `global_config.py` so resolution and rollout attribution share that rule. Benchmarks record the environment server, and `gym eval list` shows it when there is no single agent. Environment manifest validation reports that a manifest, which describes one agent, cannot describe such a taskset. `resources-server-apis.mdx` describes the route check. This supersedes NVIDIA-NeMo#4059. No issue is linked. It follows up NVIDIA-NeMo#3929, from review on NVIDIA-NeMo#4050, and provides the runtime support NVIDIA-NeMo#3800 needs. ## Validation - `tests/unit_tests/test_global_config.py`: a multi-agent server fronts all of its agents and needs no relays, removing one agent's reference makes that agent unfronted, and a composition swap retargets only the swapped agent's reference. The rename hint test runs for an `agent_server` and a `user_agent` reference. - `tests/unit_tests/test_rollout_collection.py`: attribution to `agent_server`, to a server's only agent, and to no agent for a server with several, and row validation accepting any agent of the server and rejecting others. - `tests/unit_tests/test_add_legacy_agent_environment_servers.py`: migrate-then-parse tests, which run the script and parse the result with `GlobalConfigDictParser`. They cover a rename whose source server is generated, custom-named, or `single_agent_turn_legacy`, a two-agent server the script must leave alone, renaming both agents of that server (parsed with `error_on_agent_without_environment_server: true`), and a rename in the same file as the server it would update. A test checks that the script and `environment_server_agent_refs` find the same references. - `tests/unit_tests/test_benchmarks.py`: a multi-agent taskset resolves without a pin, to none or to the server's `agent_server`; any pin on a taskset dataset is rejected; a reference to a non-agent is rejected; a declaring participant is accepted; benchmark, example, and train datasets collate and route end to end; the benchmark listing names the environment server; and manifest validation reports a multi-agent taskset. These tests fail without the change. - `pytest tests/unit_tests/test_global_config.py tests/unit_tests/test_add_legacy_agent_environment_servers.py tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_environment_scaffold.py tests/unit_tests/test_benchmarks.py tests/unit_tests/test_task_materialization.py`: all pass. - `pytest tests/unit_tests`: the tests that fail in this environment fail identically on `main`. They need the `sandbox` and telemetry extras, or Slurm. - With NVIDIA-NeMo#3800 merged locally: the script leaves `resources_servers/usersim/configs/usersim.yaml` unchanged. After renaming `usersim_assistant`, the usersim config parses with `error_on_agent_without_environment_server: true`, `usersim_environment` fronts both agents, and both usersim routes still name an existing server. - `python scripts/add_legacy_agent_environment_servers.py --check` on this repository reports the same files and blocks before and after this change. - `pre-commit run` on the changed files: all pass. ## Rollouts N/A. Single-agent servers resolve, route, and attribute exactly as before, and no configuration in this repository changes. The new behavior applies only to servers that reference agents through fields other than `agent_server`. None exist on `main` until NVIDIA-NeMo#3800, which carries its own rollout evidence. The migration script runs offline. ## Compatibility and benchmark impact - Environment servers that front several agents no longer get generated relays, and their agents route to them. - The error for a row that names the wrong agent for its environment server now lists the server's agents. - A dataset that declares `taskset:` can no longer set `agent:`. NVIDIA-NeMo#3818 accepted a pin that matched the routed agent; no configuration in this repository declares a taskset. - `gym eval list --json` adds an `environment_server` field, and `agent_name` is null for a taskset routed to a server with several agents. - The migration script updates a declared server in place when an agent is renamed, so server names and routes no longer change. - No benchmark impact. --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Co-authored-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> (cherry picked from commit 89c065b) Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>

…VIDIA-NeMo#4053) ## What does this PR do? Makes Gym and its migration tooling agree on which agents an environment server fronts, including servers that front several agents. It combines the runtime change from NVIDIA-NeMo#4059 with fixes to `scripts/add_legacy_agent_environment_servers.py`, the `ServerRefNotFoundError` rename hint, and the troubleshooting docs, so the environment server and agent server changes land together. The same commits are in the `r0.7.0` cherry-pick of NVIDIA-NeMo#3929 (NVIDIA-NeMo#4050). ## Where this fits Datasets reach an environment server in one of two ways today: run request rows route by their agent, and datasets that declare `taskset:` are materialized during collation and route by `environment_server_routes`. The goal is one path, where every dataset declares a taskset and routes to an environment server: ```mermaid flowchart LR P["Preparation script task rows with a stable task_id"] --> C["Collation validates, computes metrics, materializes"] Y["Dataset YAML file + taskset"] --> C S["task_data.py schemas environment server + resources server"] -.-> C C --> R["Rollout collection routes by taskset"] T["environment_server_routes"] --> R R --> E["Environment server single-agent, multi-agent, or legacy_agent relay"] ``` This PR is one step on the way there. Each step is additive, and datasets without `taskset:` keep working until the last one: ```mermaid flowchart TB S3818["PR 3818 (merged) collation materializes declared tasksets"] S4053["PR 4053 tasksets route to servers with several agents"] S4096["PR 4096 stable, unique task IDs"] S4097["PR 4097 metrics for every row"] S4098["PR 4098 schema per environment server"] S4099["PR 4099 legacy_agent relays serve tasksets"] SNEXT["Later configs declare taskset:, then agent routing retires"] S3818 --> S4053 --> S4096 --> S4097 --> S4098 --> S4099 --> SNEXT classDef done fill:#F1EFE8,stroke:#5F5E5A,color:#444441 classDef this fill:#EEEDFE,stroke:#534AB7,color:#26215C,stroke-width:2px classDef next fill:#E1F5EE,stroke:#0F6E56,color:#085041 class S3818 done class S4053 this class S4096,S4097,S4098,S4099,SNEXT next ``` Gray: merged. Purple: this PR. Green: other steps. ### Environment servers that front several agents An environment server can front several agents, such as the user and assistant agents of the user-simulation server in NVIDIA-NeMo#3800. It references them through fields such as `user_agent` and `assistant_agent`. Gym only recognized the `agent_server` field. For such a server: - The deprecation check from NVIDIA-NeMo#3929 treated its agents as unfronted and generated a `legacy_agent` relay in front of each. - Agent-routed rows could not find the server, because `_environment_servers_by_agent` only read `agent_server`. The first commit from NVIDIA-NeMo#4059 (authored by Felipe Vieira Frujeri) adds `environment_server_agent_refs` to `nemo_gym/global_config.py`. Any top-level field whose value has `type: responses_api_agents` references an agent, and so does `agent_server`, whose type may be left out. The deprecation check, agent routing, and `_retarget_environment_servers` (which follows an agent swapped during composition) use it. Two places in `nemo_gym/rollout_collection.py` still read only `agent_server`: - `_agent_name_for_row` names the agent a row's results are attributed to. It decides whether token IDs are captured and labels aggregate metrics. Results are now attributed to the server's `agent_server`, or to the only agent it references. A server that fronts several agents without an `agent_server` has no single agent to attribute, as before. - `_validate_environment_servers` rejects a row whose `agent_ref` names an agent the server does not front. It skipped the check for any server without `agent_server`. A row may now name any agent its server references. ### The migration script The script assumed an environment server references one agent, through `agent_server`. It now finds agent references with the same rule as `environment_server_agent_refs`. The script depends only on PyYAML, so it keeps its own copy of the rule, and a test checks that both agree. Each referenced agent counts as fronted, so the script no longer adds relays in front of agents a multi-agent server already fronts, such as those in `resources_servers/usersim/configs/usersim.yaml` from NVIDIA-NeMo#3800. For an agent renamed with `_inherit_from`, the renamed agent's server used to inherit from the name the script would generate for the source, always as a `legacy_agent` relay. That failed in three ways: - The source's server has a custom name: the inherited name does not exist, raising `InheritPathNotFoundError`. - The source's server is `single_agent_turn_legacy`: the inherited server gains a second `legacy_agent` block without an entrypoint, raising `AlmostServerError`. - Inheriting retires the source server's name, which `environment_server_routes` may still use. For example, usersim routes its tasksets to `usersim_environment`. The script now handles a rename this way: - Where a declared server references the source, the script points that reference at the new name, on the existing server. The server keeps its name, so routes and the server's other agents are unaffected. Renaming several agents of one server writes one block for that server. - Where the source has only the server the script generates, the renamed agent's server inherits it, as before. - A rename in the same file as a server that references its source cannot be migrated by appending, because the server's key would repeat. The script reports the field to update by hand and exits with 2. ### The renamed-agent hint When an environment server references an agent that no longer exists, `ServerRefNotFoundError` suggests the agent may have been renamed with `_inherit_from`. The hint said the environment server "must be renamed with it", which no longer matches the script. - The hint now says the environment server must reference the agent's new name, and names the field to update instead of always `agent_server.name`. - It is shown for any agent reference on an environment server, not only `agent_server`. - `fern/versions/latest/pages/troubleshooting/configuration.mdx` describes the same rename behavior, and the script's exit code 2 for a rename that needs a manual update. ### Declared tasksets routed to a multi-agent server NVIDIA-NeMo#3818 added `resolve_dataset_agent`'s taskset path, used by benchmark discovery, `gym eval prepare`, manifest validation, and collation. It read only the routed environment server's `agent_server`, so a taskset routed to a server that fronts several agents failed with "must bind a valid agent_server". A taskset routes to an environment server, not to an agent: rollout collection sends materialized rows to the server that `environment_server_routes` names and never uses an agent. So resolution now checks the route instead of choosing an agent: - Every agent the server references, through `environment_server_agent_refs`, must exist. - A dataset declared on an agent must be one of those agents. - A dataset declared on a resources server must be the one the server binds. - A dataset with `taskset:` cannot set `agent:`. Only benchmark datasets accept `agent:`, so requiring a pin would have left example and train datasets for a server with several agents unable to collate, and would have made an environment server that owns its dataset name one of its own agents. It returns the declaring agent, or the server's attributed agent (its `agent_server`, or its only agent), or none when the server fronts several agents without an `agent_server`. `environment_server_agent_names` and `environment_server_attributed_agent` move to `global_config.py` so resolution and rollout attribution share that rule. Benchmarks record the environment server, and `gym eval list` shows it when there is no single agent. Environment manifest validation reports that a manifest, which describes one agent, cannot describe such a taskset. `resources-server-apis.mdx` describes the route check. This supersedes NVIDIA-NeMo#4059. No issue is linked. It follows up NVIDIA-NeMo#3929, from review on NVIDIA-NeMo#4050, and provides the runtime support NVIDIA-NeMo#3800 needs. ## Validation - `tests/unit_tests/test_global_config.py`: a multi-agent server fronts all of its agents and needs no relays, removing one agent's reference makes that agent unfronted, and a composition swap retargets only the swapped agent's reference. The rename hint test runs for an `agent_server` and a `user_agent` reference. - `tests/unit_tests/test_rollout_collection.py`: attribution to `agent_server`, to a server's only agent, and to no agent for a server with several, and row validation accepting any agent of the server and rejecting others. - `tests/unit_tests/test_add_legacy_agent_environment_servers.py`: migrate-then-parse tests, which run the script and parse the result with `GlobalConfigDictParser`. They cover a rename whose source server is generated, custom-named, or `single_agent_turn_legacy`, a two-agent server the script must leave alone, renaming both agents of that server (parsed with `error_on_agent_without_environment_server: true`), and a rename in the same file as the server it would update. A test checks that the script and `environment_server_agent_refs` find the same references. - `tests/unit_tests/test_benchmarks.py`: a multi-agent taskset resolves without a pin, to none or to the server's `agent_server`; any pin on a taskset dataset is rejected; a reference to a non-agent is rejected; a declaring participant is accepted; benchmark, example, and train datasets collate and route end to end; the benchmark listing names the environment server; and manifest validation reports a multi-agent taskset. These tests fail without the change. - `pytest tests/unit_tests/test_global_config.py tests/unit_tests/test_add_legacy_agent_environment_servers.py tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_environment_scaffold.py tests/unit_tests/test_benchmarks.py tests/unit_tests/test_task_materialization.py`: all pass. - `pytest tests/unit_tests`: the tests that fail in this environment fail identically on `main`. They need the `sandbox` and telemetry extras, or Slurm. - With NVIDIA-NeMo#3800 merged locally: the script leaves `resources_servers/usersim/configs/usersim.yaml` unchanged. After renaming `usersim_assistant`, the usersim config parses with `error_on_agent_without_environment_server: true`, `usersim_environment` fronts both agents, and both usersim routes still name an existing server. - `python scripts/add_legacy_agent_environment_servers.py --check` on this repository reports the same files and blocks before and after this change. - `pre-commit run` on the changed files: all pass. ## Rollouts N/A. Single-agent servers resolve, route, and attribute exactly as before, and no configuration in this repository changes. The new behavior applies only to servers that reference agents through fields other than `agent_server`. None exist on `main` until NVIDIA-NeMo#3800, which carries its own rollout evidence. The migration script runs offline. ## Compatibility and benchmark impact - Environment servers that front several agents no longer get generated relays, and their agents route to them. - The error for a row that names the wrong agent for its environment server now lists the server's agents. - A dataset that declares `taskset:` can no longer set `agent:`. NVIDIA-NeMo#3818 accepted a pin that matched the routed agent; no configuration in this repository declares a taskset. - `gym eval list --json` adds an `environment_server` field, and `agent_name` is null for a taskset routed to a server with several agents. - The migration script updates a declared server in place when an agent is renamed, so server names and routes no longer change. - No benchmark impact. --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Co-authored-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>

…vers (#3818, #4070, #3929, #4053) into `r0.7.0` (#4107) ## What This PR only cherry-picks four already-merged `main` PRs into `r0.7.0`. It adds no new code. The commits are applied in their `main` merge order with `git cherry-pick -x`, so each commit message records its source commit. | Original PR | Commit on `main` | Change | |---|---|---| | #3818 | `325fb49d6` | feat(data): collate declared tasksets | | #4070 | `412efc730` | refactor: name episode requests and sessions without "native" | | #3929 | `1f2446c17` | feat(config): deprecate agents without an environment server and fix the migration script | | #4053 | `89c065b0e` | feat(config): support environment servers that front several agents | ## Why #4070 and #4053 carry the `r0.7.0` label, but the automatic cherry-pick workflow could not apply them: both hit merge conflicts on `r0.7.0`. The workflow still reports success, so the failure was silent. #3818 was intended for `r0.7.0` but never had the label. #3929 is included because #4053 builds on it. Every #4053 conflict came from #3929 changes to `nemo_gym/global_config.py`, `scripts/add_legacy_agent_environment_servers.py`, and their tests. With #3929 applied first, #4053 applies cleanly. ## Conflict resolution Picking #3818 first removed one #4070 conflict. The remaining conflicts were in tests that other PRs added on `main` after the branch cut: - **#4070**: renames identifiers, comments, and test names; no behavior change. Where the conflict was a test that exists only on `main` (`single_agent_turn`, SWE-bench Pro, and Hermes `test_agent_sessions.py`), the `r0.7.0` version is kept and the `main`-only test is not brought in. In `single_agent_turn_legacy`, the existing test is renamed to `test_flat_rows_and_episode_requests_project_the_same_result` to match `main`, and the `task_source` tests from a separate PR are left out. - **#3929**: one import line in `tests/unit_tests/test_global_config.py`. `main` also imports `hide_secrets_in_overrides`, added by #3679 and unused in this file, so the `r0.7.0` import is kept. - **#3818** and **#4053**: applied cleanly. ## Validation - `pytest tests/unit_tests/test_global_config.py tests/unit_tests/test_add_legacy_agent_environment_servers.py tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_environment_server.py tests/unit_tests/test_task_materialization.py tests/unit_tests/test_benchmarks.py`: pass. - `pytest environment_servers/single_agent_turn/tests` and `pytest environment_servers/single_agent_turn_legacy/tests` (run separately): pass. - `pytest tests/unit_tests -n 8`: same failures as unmodified `r0.7.0` and no new ones. The existing failures are local-environment tests in `test_opensandbox_compose.py`, `test_sandbox_stop_retry.py`, and `test_slurm_script.py`. - `pre-commit run --files` on all changed files: pass. - Checked for old names #4070 renames (`_native_request`, `_native_episode_request_body`, `_native_session_locks`, `_closed_native_sessions`): none remain. The few remaining "native row" comments are also on `main`. - SWE-bench Pro and Hermes server tests were not run locally (they need per-server venvs); release-branch CI covers them. ## Rollout evidence N/A. Each change was reviewed and validated on `main`. This PR only carries them to the release branch: #4070 is a rename with no behavior change, and the config changes are covered by the unit tests above. --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>

## Expected Harden Hermes sandbox sessions. Data/CLI changes are separate in NVIDIA-NeMo#3818. Implements [MB-1694](https://linear.app/nvidia/issue/MB-1694/runbook-onboard-sandboxed-harnesses-to-native-environmentserver). ## Added - Shared base-agent seed/close retry handling, immutable seed binding, cookie checks, session locks, and close receipts. - Confirmed sandbox process cleanup, launch fencing, internal deadlines, and owned-sandbox teardown. Close confirms cleanup before cancelling borrowed-sandbox execution. - Wall-time stops checkpoint partial results; stale PID signaling is guarded by the remote receipt. Failed setup retains cleanup-only state through a shared API. - SWE-Pro keeps its existing scoring and completion rules. Patch-extraction exceptions remain visible in the task's `error` field without masking the sample. - Identical activation retries join or replay the original run. Provider failures remain retryable and skip verification. - One request validator and prompt/settings builder for host and sandbox execution. ## Concerns - Sandbox prompt composition is unchanged: config system prompt, request instructions, then input system message. Host execution now combines these prompts instead of letting config replace the dataset prompt, and honors request temperature, including zero. `developer` messages now return 422. These host-path changes can change benchmark scores. - Unsupported non-default controls return 422, including `top_p=1.0`, `store`, `service_tier`, and metadata overrides. `model` must match the configured model. `max_output_tokens` is rejected because Hermes does not enforce a total-response budget; config `max_tokens` remains per model call. These 422s are permanent episode errors. - One activation per session, one agent worker, Linux exec-capable sandbox. Close retries must finish within the retention window. Final usage aggregation remains TODO. - Separate follow-up found during smoke: the iteration-limit summary sends `function.call_id` / `function.response_item_id` in tool history, which Gym rejects with 422. The run returns `incomplete`; its existing patch remains gradable. This update does not fix that summary path. - Simple Agent (NVIDIA-NeMo#3784) is not modified here. The SWE-Pro recipe still uses prepared flat rows through `single_agent_turn_legacy` and the EnvironmentServer session lifecycle. ## Validation Latest review follow-up: `e378cd45c` (includes the merge-queue assertion fix in `4df61bf18`). - Removed the redundant session-lock lookup guard and unused thinking-setting diagnostics. Close-requested stops now report `cancelled`, not `wall_time`; deadline behavior and scoring are unchanged. - Five targeted regression cases failed on the previous code. Final focused suite: **563 passed on Linux**; **548 passed, 15 platform skips on macOS**, Python 3.13.14. - `pre-commit run --all-files`, `git diff --check`, and **18 Fern link tests** passed. `cd fern && npm run check`: **0 errors, 1 warning**. Cleanup docs now identify close ordering, not `setsid()` alone, as the guarantee. - Real-model Hermes × SWE-Pro smoke with `nvidia/nemotron-3-super-120b-a12b` and a 15-second runner limit: 3 model calls, `incomplete` / `stop_reason=wall_time`, reward **0**, `mask_sample=false`, `evaluation_completed=true`, 18 verifier tests. Agent close preceded verification; no owned containers/network remained. Runtime file hashes matched the local revision. This is a lifecycle smoke, not an accuracy baseline. <details> <summary>Latest focused test command</summary> ```bash python -m pytest \ environment_servers/single_agent_turn/tests environment_servers/single_agent_turn_legacy/tests \ environment_servers/legacy_agent/tests tests/unit_tests/test_episode_types.py \ tests/unit_tests/test_environment_server.py tests/unit_tests/test_rollout_outcomes.py \ tests/unit_tests/test_failure_kinds.py responses_api_agents/hermes_agent/tests \ responses_api_agents/hermes_sandboxed_agent/tests resources_servers/swebench_pro/tests \ tests/unit_tests/test_agent_session_retries.py tests/unit_tests/test_agent_sessions.py \ tests/unit_tests/test_base_responses_api_agent.py tests/unit_tests/test_process_supervisor.py \ tests/unit_tests/test_local_provider.py \ --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short -rs ``` </details> Scoring revert: `f06ef8dba888fcffaf86eae265fb63f3411db5f6`. - `python -m pytest resources_servers/swebench_pro/tests environment_servers/single_agent_turn/tests environment_servers/single_agent_turn_legacy/tests --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short`: **135 passed**. Four failure-policy regression cases failed before the revert, then passed; extraction errors retain the original exception message. - `pre-commit run --all-files` and `git diff --check` passed. - Real-model SWE-Pro HTTP smoke with `nvidia/nemotron-3-super-120b-a12b`, a 15-second agent limit, and an invalid Git base injected only in the smoke driver: reward **0**, `mask_sample=false`, `evaluation_completed=true`, and the actual extraction exception in `error`. The agent closed before verification; all owned containers/network were cleaned up. This checks failure reporting, not benchmark accuracy. - No documentation change needed: this restores the existing verifier contract rather than adding an API. Earlier review-fix validation at `95a2b6481fa0225fc5a363443d9a536e98cd304f`: - macOS/Python 3.13.14: **494 passed, 13 platform skips**. - Linux/Python 3.13.14: **422 passed** across Hermes, shared sessions, process supervision, SWE-Pro, and Environment Server tests. Final verifier tests: **49 passed**; final supervisor/runner recheck: **40 passed**. - Real-process regressions cover provider-group cancellation, lost spawn handles, detached children, timeout checkpoints (including an uncooperative worker), stale-PID protection, partial setup, and cleanup retries. - Two real-model SWE-Pro HTTP smokes with `nvidia/nemotron-3-super-120b-a12b`: both reached verification after agent close, returned valid reward 0, and left no owned containers or networks. One stopped on model-output truncation; the forced 15-second wall-time run returned `incomplete` / `stop_reason=wall_time`. Both had no patch; these are lifecycle checks, **not accuracy baselines**. Partial-work preservation is separately covered by deterministic subprocess tests. - `pre-commit run --all-files` and diff checks passed; Fern: 0 errors, 1 warning. The test import conflict with main was removed without rebasing. - Full repository CI is not claimed. The inherited DCO mismatch in `07ca331ca1e12d234f1a3a60bb12e1651e496786` remains; the new commit is signed off. <details> <summary>Current focused command</summary> ```bash python -m pytest responses_api_agents/hermes_agent/tests \ responses_api_agents/hermes_sandboxed_agent/tests \ tests/unit_tests/test_process_supervisor.py tests/unit_tests/test_local_provider.py \ tests/unit_tests/test_sandbox_stop_retry.py tests/unit_tests/test_base_responses_api_agent.py \ tests/unit_tests/test_agent_sessions.py tests/unit_tests/test_agent_session_retries.py \ tests/unit_tests/test_environment_server.py resources_servers/swebench_pro/tests \ environment_servers/single_agent_turn/tests \ --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short python -m pre_commit run --all-files cd fern && npm run check ``` </details> <details> <summary>Previous validation at d5f13f5 (historical)</summary> Head: `d5f13f54cf762c7e687dd2d50d2583efe2c9b717`. - macOS/Python 3.13.14: **487 passed, 8 skipped** (Linux-only checks). - Linux/Python 3.13.14: **292 passed**, including Hermes, sessions, process cleanup, and EnvironmentServer tests. Uploaded source and test hashes match this commit. - New regressions failed against the old code: ignored controls and host/sandbox prompt/settings drift. Also test matching model names, zero temperature, unchanged input, validation once per request, and rejection of future non-default schema controls. - `pre-commit run --all-files`, diff checks, and Fern passed (0 errors, 1 warning). New commit is DCO-signed. - Fresh real-model public SWE-Pro HTTP smoke using `nvidia/nemotron-3-super-120b-a12b`: **reward 1.0**, **18/18 verifier tests passed**, 11,273-byte patch. Verified prompt ordering and temperature at the Model Server, agent close before verification, Resources close, and no remaining owned containers/network. Inspected the patch and verifier results. Agent reached the 20-turn smoke limit and returned `incomplete`, with the separate summary-call 422 described above. - This is one functional smoke, not an accuracy/baseline comparison. Reused isolated dependency environments; full repository CI is not claimed. The inherited DCO author/sign-off mismatch in `07ca331ca1e12d234f1a3a60bb12e1651e496786` is unchanged. <details> <summary>Focused commands</summary> ```bash python -m pytest responses_api_agents/hermes_agent/tests \ tests/unit_tests/test_agent_session_retries.py \ tests/unit_tests/test_agent_sessions.py tests/unit_tests/test_base_responses_api_agent.py \ tests/unit_tests/test_process_supervisor.py tests/unit_tests/test_local_provider.py \ responses_api_agents/hermes_sandboxed_agent/tests \ environment_servers/single_agent_turn/tests environment_servers/single_agent_turn_legacy/tests \ resources_servers/swebench_pro/tests tests/unit_tests/test_resources_sessions.py \ tests/unit_tests/test_environment_server.py \ --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short -rs # Linux subset: python -m pytest tests/unit_tests/test_process_supervisor.py tests/unit_tests/test_local_provider.py \ tests/unit_tests/test_agent_session_retries.py responses_api_agents/hermes_agent/tests \ environment_servers/single_agent_turn/tests \ --import-mode=importlib -q -o addopts='' -o log_cli=false --tb=short -rs python -m pre_commit run --all-files cd fern && npm run check ``` </details> </details> --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com> Signed-off-by: Adil Asif <adasif@nvidia.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-authored-by: Ananth Subramaniam <ansubramania@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <martas@nvidia.com> Co-authored-by: Adil Asif <adasif@nvidia.com> (cherry picked from commit b846dfa) Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>

…ions (#3554, #3784) into `r0.7.0` (#4113) ## What This PR only cherry-picks two already-merged `main` PRs into `r0.7.0`. It adds no new code. Each commit is applied with `git cherry-pick -x`, so its message records the source commit. | Original PR | Commit on `main` | Change | |---|---|---| | #3554 | `b846dfaa5` | fix(hermes): harden sandbox sessions | | #3784 | `e0e63e526` | feat(simple-agent): run simple agent through environment server sessions | This branch is based on the current `r0.7.0`, which already includes #4107 (#3818, #4070, #3929, #4053) and #4078. ## Why #3784 has the `r0.7.0` label, but the automatic cherry-pick workflow could not apply it: it hit a merge conflict in `nemo_gym/base_responses_api_agent.py` and reported success anyway, so nothing was opened. #3554 is included because #3784 builds on it. #3554 added the agent session handling (seed and close retries, session locks, close receipts) in `nemo_gym/base_responses_api_agent.py` that #3784 extends. With #3554 applied first, the conflict in that file goes away. #3554 has no `r0.7.0` label; it is included on purpose. ### Behavior changes from #3554 #3554's own description lists these, and they now apply on `r0.7.0` too: - Hermes host execution now combines the config system prompt, request instructions, and input system message instead of letting the config prompt replace the dataset prompt. It honors the request temperature, including zero, and rejects `developer` messages with 422. These changes can change benchmark scores for Hermes on the host path. - Hermes rejects unsupported non-default controls with a permanent 422, including `top_p=1.0`, `store`, `service_tier`, metadata overrides, `max_output_tokens`, and a `model` that does not match the configured model. ## Conflict resolution - **#3554**: applied cleanly. - **#3784**: one conflict in `environment_servers/single_agent_turn_legacy/tests/test_app.py`. #3784 adds two `task_source` tests next to a test that #4070 (already on `r0.7.0` through #4107) renamed. The resolution keeps both, and the new tests call the adapter method by its post-#4070 name, `_episode_request_from_row`. The resulting file is identical to the file on `main`. ## Validation - `pytest tests/unit_tests -n 8`: same failures as unmodified `r0.7.0` and no new ones. The existing failures are local-environment tests in `test_opensandbox_compose.py`, `test_sandbox_stop_retry.py`, and `test_slurm_script.py`. - #3554's focused suite, each path run separately with `--import-mode=importlib`: `environment_servers/single_agent_turn/tests`, `environment_servers/single_agent_turn_legacy/tests`, `environment_servers/legacy_agent/tests`, `responses_api_agents/hermes_sandboxed_agent/tests`, `resources_servers/swebench_pro/tests`, and `tests/unit_tests/test_{episode_types,environment_server,rollout_outcomes,failure_kinds,agent_session_retries,agent_sessions,base_responses_api_agent,process_supervisor,local_provider}.py`: all pass. - `responses_api_agents/hermes_agent/tests` in its own venv from `requirements.txt`: all pass. - `responses_api_agents/simple_agent/tests` and `resources_servers/example_single_tool_call/tests`: all pass. - `pre-commit run --files` on all files changed by the two commits: pass. ## Rollout evidence N/A for the cherry-pick itself. Both changes are applied unchanged from `main`, except for the one test-file conflict described above. #3554 includes a real-model Hermes with SWE-bench Pro lifecycle smoke rollout. --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com> Signed-off-by: Adil Asif <adasif@nvidia.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Co-authored-by: Marta Stepniewska-Dziubinska <martas@nvidia.com> Co-authored-by: Adil Asif <adasif@nvidia.com>

Integrate current main and use the production metrics serializer in the committed-statistics regression. Exclude taskset routing metadata, matching NVIDIA-NeMo#3818, and omit materialized-task-only statistics for Responses-style datasets, matching NVIDIA-NeMo#4060. Keep the existing fixture and full equality assertion intact. Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

…A-NeMo#4131) ## What does this PR do? Adds `gym eval plan`, which prepares one benchmark and writes a **plan file**: every task once, sorted by task id, with a SHA-256 digest of the exact input a run receives. **Why:** Gym has no single command that answers "which tasks does this benchmark have, and with exactly what inputs?" in a stable form. The prepared JSONL depends on preparation details, its row order is not guaranteed, and many rows have no id, so collation falls back to the row position. `gym eval plan` produces that answer as one file: stable task ids, a digest of each task's exact run input, and a versioned, documented format. The plan stores ids and digests, not the inputs themselves, so it stays small; a run re-prepares the data, and the digests show whether it still matches the plan. What the command does: - Selects the benchmark with the existing `--benchmark NAME[/FLAVOR]` lookup and runs its `prepare.py` (shared code with `gym eval prepare`). - Takes each task's id from `task_id`, `problem_id` or `instance_id`, the same precedence as `materialize_task` (NVIDIA-NeMo#3818); the field list is now defined once, as `TASK_ID_FIELDS` in `nemo_gym/task_materialization.py`, and used by `materialize_task`, rollout collection and the plan. Unlike collation, **there is no fallback to the row position**: a position-based id names a different task once rows are reordered, so benchmarks without an id field are refused with an actionable error. - Each task is `{task_id: {taskset, task_id}, digest}`. The digest is `sha256(canonical JSON of task_input)`, where `task_input` is the `MaterializedTask` input with routing and bookkeeping fields (`agent_ref`, `task_source`, `_ng_*`) excluded. The taskset is the dataset's declared `taskset`, or the benchmark name when it declares none. - Records the Gym version and commit and the selected config paths (relative to the checkout). - Output is deterministic (sorted tasks, canonical JSON): same Gym commit and same prepared data give a byte-identical file. - `--task-ids` plans a subset; `--schema` prints the JSON Schema; the format is labelled `nemo-gym.eval-plan.v1` and is meant to change only additively within that version. - Benchmarks with a prompt template (`prompt_config`) are refused for now; applying the template is a follow-up. `prepare_benchmark` is split into `_datasets_to_prepare` and `_prepare_datasets` so both commands share them; its behaviour is unchanged, except that the "no preparable dataset" error no longer names `gym eval prepare` (it suggests `--config <config path>` or `--benchmark <name>`). The plan considers only the benchmark datasets among the datasets to prepare. No issue filed: this is a new, additive command. Related: the Harbor task work (NVIDIA-NeMo#3673 and follow-ups) produces the same `MaterializedTask` rows and pins, so plans for Harbor datasets should line up with it. Feedback on the file format is very welcome while this is a draft. ## Validation - `pytest tests/unit_tests/test_eval_plan.py`: 20 passed (id precedence, ids and digests only (no inputs) in the file, declared taskset used, refusal of missing and duplicate ids, refusal of already-materialized rows as a config error, routing fields excluded from input and digest, subset selection, deterministic output, schema round trip, CLI: file written, subset, prompt-template refusal, several-benchmarks refusal, `--out` required, `--schema`). - `pytest tests/unit_tests/test_cli_main.py tests/unit_tests/test_cli_compat.py tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_task_materialization.py tests/unit_tests/test_benchmarks.py`: all pass after rebasing onto `main`, except `test_benchmarks.py::test_every_repo_benchmark_appears_in_listing`, the DNS `gaierror` on my machine noted below. - `pytest tests/unit_tests/test_benchmarks.py tests/unit_tests/test_cli_main.py tests/unit_tests/test_cli_compat.py`: 335 passed, 1 failed. The failure (`test_every_repo_benchmark_appears_in_listing`) is a DNS `gaierror` while resolving four configs on my machine and fails identically on unmodified `main`. - `pre-commit run --all-files`: all hooks pass. - Real runs: - `gym eval plan --benchmark swebench/verified/opencode --out plan.json`: 500 tasks keyed by `instance_id`; running it twice gives byte-identical files. - `--task-ids astropy__astropy-12907,django__django-11099`: plan with those 2 tasks. - `--benchmark terminal_bench_2_1/terminus_2` (fresh preparation, on `main` with NVIDIA-NeMo#4135): 89 tasks, keyed by task folder name. - `--benchmark aime24`: refused because of its prompt template. ## Rollouts N/A: the command only prepares data and writes a file; no server, agent or model code changes. ## Compatibility and benchmark impact None. New command; `gym eval prepare` behaviour unchanged. ## Checklist - [x] I have read the [contributing guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Documentation added or updated, or N/A with justification. - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format checks pass). - [x] New source files include the required Apache-2.0 SPDX header. - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Ingo Guehring <iguehring@nvidia.com>

yaoyu-33 added feature New capabilities, enhancements, or enablement work area:data Datasets, preparation, task formats, and materialization labels Sep 30, 2026

yaoyu-33 mentioned this pull request Sep 30, 2026

fix(hermes): harden sandbox sessions #3554

Merged

yaoyu-33 marked this pull request as ready for review September 30, 2026 19:43

yaoyu-33 added complexity:medium Single-domain change with interacting parts or a moderate review surface needs-review PR is ready for code review and waiting on a reviewer labels Sep 30, 2026

ananthsub reviewed Oct 1, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to public October 2, 2026 01:00 Inactive

nemo-automation-bot Bot requested review from a team October 2, 2026 01:00

copy-pr-bot Bot temporarily deployed to public October 2, 2026 01:02 Inactive

copy-pr-bot Bot temporarily deployed to public October 2, 2026 01:03 Inactive

ananthsub reviewed Oct 2, 2026

View reviewed changes

Comment thread nemo_gym/config_types.py Resolved

Comment thread nemo_gym/config_types.py Resolved

yaoyu-33 mentioned this pull request Oct 2, 2026

docs(review): add contract-focused review guidance #3954

Draft

ananthsub reviewed Oct 5, 2026

View reviewed changes

Comment thread nemo_gym/single_agent_turn_task.py Resolved

ananthsub reviewed Oct 5, 2026

View reviewed changes

Comment thread nemo_gym/train_data_utils.py Resolved

yaoyu-33 changed the title ~~feat(data): collate declared single-agent tasksets~~ feat(data): collate declared tasksets Oct 5, 2026

ananthsub reviewed Oct 5, 2026

View reviewed changes

Comment thread nemo_gym/train_data_utils.py Resolved

Comment thread nemo_gym/single_agent_turn_types.py Resolved

Comment thread tests/unit_tests/test_task_materialization.py Resolved

nemo-automation-bot Bot requested review from a team October 5, 2026 13:31

copy-pr-bot Bot temporarily deployed to public October 5, 2026 13:31 Inactive

copy-pr-bot Bot temporarily deployed to public October 5, 2026 23:11 Inactive

copy-pr-bot Bot temporarily deployed to public October 5, 2026 23:12 Inactive

ffrujeri added this pull request to the merge queue Oct 6, 2026

Merged via the queue into NVIDIA-NeMo:main with commit 325fb49 Oct 6, 2026
43 checks passed

ananthsub mentioned this pull request Oct 6, 2026

cp: feat(config): deprecate agents without an environment server and fix the migration script (3929) into r0.7.0 #4050

Closed

ananthsub mentioned this pull request Oct 6, 2026

cp: Hermes session hardening and simple agent environment server sessions (#3554, #3784) into r0.7.0 #4113

Merged

iguehring-nvidia mentioned this pull request Oct 6, 2026

feat(eval): add gym eval plan to write a benchmark's task list #4131

Merged

7 tasks done

yuhezhang-ai mentioned this pull request Oct 6, 2026

Katherine/deepswe external1 #3597

Merged

zyzhou5 mentioned this pull request Oct 6, 2026

fix(config): preserve environment server references when agents are renamed #4141

Merged

ananthsub mentioned this pull request Oct 8, 2026

feat(data): compute dataset metrics for every row #4097

Draft

Conversation

yaoyu-33 commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Expected

Added

Compatibility and benchmark impact

Validation

Rebase onto current main (2026-10-05)

Completed CI on the rebased head (2026-10-05)

Historical validation before the rebase

Uh oh!

copy-pr-bot Bot commented Sep 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yaoyu-33 commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ananthsub commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yaoyu-33 commented Sep 30, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading