Repository navigation
Conversation
## Summary - Snapshots `fern/versions/latest/` → `fern/versions/v0.5.0/` (101 files) to freeze the docs at the 0.5.0 GA state - Creates `fern/versions/v0.5.0.yml` with nav config pointing to the frozen pages - Registers `v0.5.0` as `stable` in `fern/docs.yml` (between Main and 0.4.0) - Adds v0.5.0 release notes accordion to `release-notes.mdx` in both `latest/` and the snapshot - Drops pre-release tag: `PRE_RELEASE = "rc0"` → `""` in `nemo_gym/package_info.py` (version is now `0.5.0`) ## Test plan - [ ] Verify `docs.nvidia.com/nemo/gym` shows `0.5.0` in the version picker as stable - [ ] Verify the v0.5.0 docs snapshot renders correctly at `/v0.5.0/` - [ ] Verify release notes page shows v0.5.0 accordion open by default, v0.4.0 collapsed - [ ] Verify `nemo_gym.__version__` returns `"0.5.0"` (no `rc0` suffix) - [ ] Cherry-pick to `r0.5.0` after merging to main 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Kajal Jain <kajalj@nvidia.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
## Summary Fixes the `test_internal_pages_are_linked_by_path_not_by_absolute_url` test failure introduced by #2389. The v0.5.0 release notes sandboxing section contained a `docs.nvidia.com/nemo/gym/...` absolute link. The test requires all internal docs links in `latest/` to use relative paths (to keep readers within the version they're reading). Replaced with a relative path `/nemo-gym/nemo_gym/sandbox/providers` in both `latest/` and the `v0.5.0/` snapshot. ## Test plan - [ ] `test_internal_pages_are_linked_by_path_not_by_absolute_url` passes - [ ] Cherry-pick to `r0.5.0` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Kajal Jain <kajalj@nvidia.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
## Summary - Remove the NeMo Evaluator row from the Evaluation & Training table on About → Ecosystem ## Test plan - [x] `make docs-check` - [x] Confirm Fern preview on the PR - [x] Verify Ecosystem page no longer lists NeMo Evaluator under Evaluation & Training Signed-off-by: Seph Mard <smard@nvidia.com>
## Summary - Adds v0.5.0 release highlights to the README News section with a link to the full GitHub release notes - Moves v0.4.0 and v0.3.0 into the Previous News collapsible as condensed one-liners 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Kajal Jain <kajalj@nvidia.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Looks like having more `num_workers` is beneficial in general. Suggesting increasing the number of HTTP front ends. Profiling run config from @sdevare-nv (nano opencode swe run) shows that the model call latency is bottlenecked by the `num_workers` in the given config. I am adding two lines of logic change for this. I think session-based routing should be adjusted by using session ID hashing when there are multiple workers in the run. Summarized the profiling results in [summary.html](https://sc.talos.nvidia.com/view/home/scratch.youngeunk_gpu_3/htmls/opencode_vs_openhands/interactive_opencode_openhands_trace_comparison.html) --------- Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
## Summary - Bumps `PATCH` from `0` to `1` in `nemo_gym/package_info.py` (cherry-pick of `0f9c375a1` from `r0.5.0`) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Taken from #2175 --------- Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Bumps the pinned SHA for `_claude_review.yml` to FW-CI-templates `v1.8.5` (`fe0ff3bbe10790c51273ee3c1189b3c0497bf860`). **What changed in v1.8.5:** - fix: tolerate trailing whitespace in `/claude review` trigger phrase cc @KajalJ
## Summary
- Installs `enroot` via the NVIDIA CUDA apt repo (same pattern used in
the RL Dockerfile for `nsight-systems-cli`) so
`nemo_gym.sandbox.providers.enroot.EnrootProvider` works out of the box
— the provider calls `shutil.which("enroot")` at startup and raises
`RuntimeError` if the binary is missing
- Adds `bash` explicitly to the `apt-get install` block for clarity
(present in the Ubuntu 24.04 base but was undeclared)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…ATH (#2428) `start_vllm_server` sets `runtime_env.py_executable` to `ray_worker_py_executable`, which gives the vLLM actor this server's venv interpreter. It does not put that venv's `bin` directory on the actor's PATH, so console scripts installed next to the interpreter are not resolvable in the actor or in the child processes it spawns. vLLM declares `ninja` as a runtime dependency and shells out to it when compiling kernels, so spinup fails with `Worker failed with error '[Errno 2] No such file or directory: 'ninja''` even though ninja is installed in the server venv. This only shows up where ninja is not already on the system PATH. The `vllm/vllm-openai` images have it at `/usr/local/bin`, so the gap was invisible there. NeMo-RL containers have ninja only inside a venv, and `local_vllm_model` does not start there at all. I reproduced it on vllm 0.24.0 and 0.25.1, so it is not tied to a version bump. ## Change `_ray_actor_path` derives the PATH from `ray_worker_py_executable`, and `start_vllm_server` passes it in `runtime_env.env_vars`. It is listed ahead of the server's own `env_vars` so a config can still override PATH. Deriving it from `ray_worker_py_executable` rather than hardcoding a directory keeps it correct wherever the venv lands, and makes `py_executable` and PATH consistent instead of the actor getting the interpreter from one venv and its binaries from another. It also covers future vLLM shell-outs rather than ninja specifically. ## Testing Three unit tests in `responses_api_models/local_vllm_model/tests/test_app.py` cover the helper (leading entry, inherited PATH preserved, no empty segment when PATH is unset), and one covers the wiring by capturing the actor's `runtime_env` — including that a server-supplied PATH still wins. End to end in `nvcr.io/nvidian/nemo-rl:nightly`, serving `Qwen/Qwen3-0.6B` via `ng_run`: without this change spinup fails on ninja; with it the server reaches `All 1 / 1 servers ready!` and returns completions Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
… rollouts (#2111) ## Summary Adds a Gym-native pipeline for generating and evaluating long-horizon synthetic conversational tool-use tasks. The pipeline is split into five composable components: - `conversational_tool_use_domain_generation`: generates domain candidates. - `conversational_tool_use_policy_tool_generation`: generates, refines, validates, and judges domain policies and tool schemas. - `conversational_tool_use_scenario_generation`: generates customer scenarios for accepted policy/tool artifacts. - `conversational_tool_use_agent`: runs the policy-model conversation loop. - `conversational_tool_use_simulation`: simulates users and tool results, maintains session state, and verifies completed trajectories. Each component uses standard Gym agents, resource servers, model-server references, rollout JSONL, and explicit materialization steps. This PR does not change `nemo_gym` core infrastructure. ## Design ### Generation pipeline The generation stages are independent Gym agents connected through typed rollout artifacts: ```text domain rollouts -> policy/tool inputs and rollouts -> scenario inputs and rollouts -> conversational tool-use inputs -> policy agent + rollout simulation ``` Each rollout preserves the generated artifact, source lineage, model requests and responses, parsing outcomes, and validation or judgment results. Materializers produce stable IDs and the exact row contract required by the next stage. The pipeline supports `general` and `proactive` profiles. Prompt assets, schemas, golden policy/tool references, checked-in examples, and package-specific configuration live with the component that owns them. ### Conversational rollout environment The policy agent supports multi-turn assistant messages, sequential and parallel function calls, prefilled histories, transfers, and bounded policy-model steps. A normal assistant message does not terminate the rollout: the agent asks the simulator for the next user turn and continues until a stop condition, transfer, incomplete response, verification failure, or step limit. The simulation resource server owns: - per-rollout policy, tools, scenario, and conversation state; - user and tool-result simulation; - generated tool routing and JSON Schema validation; - message-level and final-conversation verification; - retryable provider-failure handling and failure labels; - cleanup of completed or abandoned sessions. Policy, user-simulator, tool-simulator, and judge roles are separate model-server references and can use different models or providers. ### Transcript and data contracts The returned Gym row has two aligned views: - `responses_create_params` and `response` contain the policy-visible Responses API transcript; - `result.trajectory` contains simulator-native user, agent, tool, terminal, and verification state. The agent validates that the materialized system prompt and model-visible tools match the top-level policy and simulator tool definitions before starting a rollout. Provider output order is preserved when a response contains assistant content and function calls. Parallel tool results are appended only after all selected response items. Each decomposed trajectory message records its source `response_id` and `response_output_index`; the complete raw response is stored once rather than duplicated on every message. Transient seeding, generation, and verification failures use Gym's failure-sidecar contract and are excluded from scored rollout JSONL. Semantic failures remain verified rollout results with synchronized reward, invalid-reason, failure-label, and judge-diagnostic fields. ### Dataset preparation The resource-server package includes builders for checked-in examples, full `general` and `proactive` datasets, combined datasets, parallel-tool-call variants, and deterministic balanced sampling. Builders reject malformed or incomplete tools, invalid JSON Schemas, malformed scenarios, duplicate tool names, and model-template leakage. Full generated datasets, reports, and rollout outputs are intentionally not checked in. ## Validation ### Automated validation - `68 passed`: `resources_servers/conversational_tool_use_simulation` test suite. - `34 passed`: `responses_api_agents/conversational_tool_use/simulation` test suite. - Ruff and repository pre-commit hooks pass. ### Live rollout validation - A hosted-inference rollout completed through Gym's `inference_provider` using DeepSeek V4 Pro as the policy and Nemotron 3 Ultra as simulator and judge. - The reward-1 trajectory exercised parallel function calls and confirmed converted output ordering as `reasoning -> message -> function_call(s)`. ### End-to-end NeMo RL validation The materialized conversational tool-use dataset was used to train `NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` through Step 30 with multi-turn GRPO. Steps 10, 20, and 30 were converted from DCP to Hugging Face format and evaluated alongside the base checkpoint on TAU2 and TAU3 BM25 artificial analysis. | Checkpoint | Airline | Retail | Telecom | TAU2 average | TAU3 BM25-AA | | --- | ---: | ---: | ---: | ---: | ---: | | Base | 33.50% | 62.94% | 44.08% | 46.84% | 4.95% | | Step 10 | 36.00% | 62.39% | 43.53% | 47.31% | 4.33% | | Step 20 | 36.00% | 63.93% | 44.08% | 48.00% | 5.36% | | Step 30 | 35.50% | 62.94% | 46.49% | 48.31% | 4.54% | `TAU2 average` is the arithmetic mean across airline, retail, and telecom. This run validates the complete downstream path from dataset materialization through rollout collection, verification, training, checkpoint conversion, and Gym evaluation. TAU2 average increased from `46.84%` at base to `48.31%` at Step 30. --------- Signed-off-by: Junkeun Yi <jkyi@nvidia.com> Co-authored-by: Jonathan Li <jonli@nvidia.com>
…kends (#2418) ## Summary Makes OpenSandbox `exec` resilient to two failure modes that both surface as proxy `502`s. 1. **execd bind race.** Sandbox pods have no readiness probe, so the server can mark a sandbox `Running` and proxy exec requests before its exec daemon (`execd`) has bound its port. The proxy then returns `502 "could not connect to the backend"` for a short window after a pod first reports `Running` (worse on cold images). The command never reached `execd`, so retrying it is safe **even when `command_retries=0`** — the default, since agent commands are usually mutating and must not double-run. 2. **Dead backend.** A container that OOM-kills mid-run leaves a live server record whose every request `502`s forever, because sandbox pods never restart. With `command_retries=0`, either mode currently fails the exec outright. ## Changes - Add `_submit_command`, which retries **only** connect-class `502`s under `operations.retries` (independent of `command_retries`) to cover the bind window. - Once that budget is exhausted, raise a typed `SandboxBackendUnreachableError` so a dead backend **fails fast** instead of retrying for hours. - Retry logging now carries `operation` + `sandbox_id`, so an absorbed create-probe retry is distinguishable from a failing agent exec. No config change: `command_retries` stays `0`, and the `502`-retry lives in a separate budget so mutating commands are never double-run. ## Testing Two unit tests in `tests/unit_tests/test_opensandbox_provider.py`: - transient `502`s are retried with `command_retries=0` without the command double-running; - persistent `502`s raise the typed backend-unreachable error. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Hemil Desai <hemild@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
## What changed - Restore redirects from the former flat `/main/training-tutorials` path to `/main/tutorials/training-tutorials`. - Restore redirects from the former flat `/main/evaluation-tutorials` path to `/main/tutorials/evaluation-tutorials`. - Cover both each tutorial root and all descendant pages with explicit root and `:path*` rules. ## Why The Training Tutorials and Evaluation Tutorials sections gained a `/tutorials` parent in the Fern navigation hierarchy. The compatibility redirects for the previous versioned paths were later lost during Fern multi-source configuration changes, causing previously shared URLs such as `/nemo/gym/main/training-tutorials/nemo-rl-grpo/` to return 404. The existing MDX links are intentionally unchanged: Fern renders them through the working unversioned redirect chain. ## User impact Previously shared `main` tutorial URLs will redirect to their current nested locations instead of returning 404. ## Validation - `make docs-check` — 0 errors (25 existing warnings) - `git diff --check` Signed-off-by: Lawrence Lane <25370251+lbliii@users.noreply.github.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>
…2 image check (#2440) ## What Adds startup validation to `PinchBenchAgentConfig` and hardens the agent's error handling and test coverage. ## Changes **`app.py`** - `web_search_provider` is now `Literal["brave", "tavily"]` — invalid values rejected at startup - Validates that the selected provider's API key is set (`_validate_api_key_present`) - Validates `sandbox_work_base` is an absolute path at startup (was only checked inside `_run_in_apptainer_direct`) - Validates `max_tokens <= context_window` - Validates `sandbox_provider` name is known via `get_provider_class` (catches typos before the first task) - **FEP-1322**: validates `sandbox_spec.image` exists on disk when it is a local path — fails fast at startup instead of silently scoring 0 across every task in a chained Slurm job - Non-clean apptainer exit (rc ≠ 0 but archive present): logs a warning and records `non_clean_exit_rc` in `raw_rollout` instead of silently continuing - `_parse_result`: logs a warning when multiple result JSON files are found - Imports `NG_*` constants from `nemo_gym.rollout_collection` instead of redefining them - Removes redundant comments, section dividers, and PR references throughout **`benchmarks/pinchbench/config.yaml`** - Minor cleanup **Tests** (`test_app.py`, `test_sandbox_isolation.py`, `test_validation.py` — new file) - 63 tests total, up from 26 - Parametrized routing sentinel tests and `_classify_task_failure` - Removed ghost assertion for `PINCHBENCH_FORCE_LOCAL`, strengthened `OPENCLAW_GATEWAY_TOKEN` check - New cases: signal-kill detection (rc −15/137/143), `non_clean_exit_rc` end-to-end, missing `task_id`, optional env var injection, `toolResult` details fallback, transcript dedup, malformed JSON toleration, image existence check --------- Signed-off-by: plaszkiewicz <plaszkiewicz@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary - add a copy-paste model call capture workflow from environment startup through inspecting `ng_model_call_capture` - show representative attachment and CaptureStore JSONL output, including gaps and the full metrics shape - lead with the consumer workflow and move custom-agent correlation helpers into an advanced section - clarify that agent observations are currently emitted by the Claude Code agent - add model call capture to the model-server landing-page card grid - keep the `latest` and `v0.5.0` documentation copies aligned ## Why Addresses NVBug 6534797. The existing page described capture internals but did not show readers how to enable capture in a real Gym run, where to find the result, or the complete output shape. ## Validation - `pytest -q tests/unit_tests/test_fern_docs_links.py` - `ruff check tests/unit_tests/test_fern_docs_links.py` - `git diff --check` Release cherry-pick companion: #2416 --------- Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>
Fixes #2436 ## Problem The Responses API can emit output-call items that were missing from `NeMoGymResponseInputItem`. Because `NeMoGymResponseOutputItem` shares that union, valid upstream responses containing those items failed local validation and the model proxy returned HTTP 500 after the provider had succeeded. ## Changes - Add SDK-backed wrappers for file search, web search, computer use, image generation, code interpreter, local shell, and custom tool calls. - Preserve each upstream model's required `type` discriminator. - Reject untagged output-call dictionaries before permissive union coercion can misclassify them. - Treat every non-message model output item as a transcript output boundary. - Keep input-only transcripts on the input side and preserve tool-only model output during simulation canonicalization. - Add regression coverage for schema validation, all output boundary types, input-only transcripts, and tool-only responses. Web search, file search, image generation, and code interpreter are provider-executed. Computer actions, local-shell commands, and custom calls are returned for the integrator to execute. ## Behavior change: the no-match case in `split_responses_input_output_items` Separately from the new item types, this changes what happens when no output boundary is found at all. Calling it out explicitly since it is not implied by the title. The old loop read the leaked loop variable after `break`, so when nothing matched it fell through with `i == len(items) - 1` and the final item was always classified as output. The new code initializes `split_at = len(items)`, so a transcript with no output boundary is now treated as entirely input. | Items | Before | After | | --- | --- | --- | | `[function_call_output]` | `([], [function_call_output])` | `([function_call_output], [])` | | `[user_message, function_call_output]` | `([user_message], [function_call_output])` | `([user_message, function_call_output], [])` | | `[user_message, function_call]` | `([user_message], [function_call])` | unchanged | The old result was an off-by-one: a trailing user or tool item was reported as model output even though the transcript contained no model turn. Transcripts that do contain a boundary item are unaffected, because the loop still breaks at the same index. Worth noting that a single-item transcript of one of the newly recognized types, for example `[web_search_call]`, returns the same pair before and after, because the old fallback happened to land on the correct split for that case. The two changes mask each other there, which is why the existing tests did not surface the difference. ### Effect on the two callers In both callers `output_items` becomes `response.output`, so the only transcripts affected are those with no output boundary anywhere. - `responses_api_agents/tau2/app.py`: the first split breaks at the leading assistant message, per the comment there, so only the second split over the remainder is in scope. If that remainder is entirely user and tool items, its trailing item is no longer reported as model output. An empty `output_items` slices safely, so `output_items[:1]` and `output_items[1:]` still behave. - `responses_api_agents/swe_agents/app.py`: only the non-replay branch calls this, since the replay branch computes its own `split_at`. A trajectory with no model turn now yields an empty `output` rather than one mislabeled trailing item. In both cases the new result is the accurate one, and neither path raises on an empty list. ## Verification - `1580 passed, 369 deselected, 5 subtests passed` in the complete non-sandbox unit suite. - `117 passed` across the response schemas, converter, model proxy, and conversational simulation suites. - All applicable pre-commit hooks passed. - `git diff --check` passed. - Live forced web-search requests returned valid `web_search_call` output for `gpt-4.1-mini`, `gpt-5.4-mini`, `gpt-5.6-luna`, and `gpt-5.6-terra`; the proxy failure was reproduced before the schema fix and resolved afterward. --------- Signed-off-by: Atharva-Kanherkar <142440039+Atharva-Kanherkar@users.noreply.github.com>
## Summary Add OpenClaw rollout observations for both the standalone agent endpoint and the PinchBench benchmark path. OpenClaw is the observation producer in both paths. PinchBench exports retained OpenClaw session artifacts and attaches the resulting OpenClaw observation bundle; it does not define a separate observation format or producer identity. Observations are additive and are emitted only when observability and rollout correlation are enabled. Existing agent responses, rewards, and grading fields are unchanged. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 are evaluated on the correlated Gym Model Server path. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | OpenClaw | V | V | X | O | V | V | V | C3 remains unsupported because OpenClaw does not emit standardized per-turn records. C4 is partial because retained branched sessions are reported but cannot always reconstruct a single model-visible branch. ## Changes - Normalize retained OpenClaw session records into user, assistant, reasoning, tool-call, and tool-result observations. - Reconstruct subagent lineage from retained OpenClaw session stores. - Correlate policy and judge model calls through rollout-prefixed Gym Model Servers when configured. - Preserve independent tool-call timing, status, duration, and output. - Record context compaction and explicit observation gaps. - Preserve direct model endpoint support when no Gym Model Server is configured. - Isolate observation capture failures from agent responses and benchmark results. ## Evidence boundaries - Model-call correlation requires transcript response IDs and a configured Gym Model Server. - OpenClaw does not identify the exact tool call that spawned a child session. - Branches within one session are reported but are not reconstructed as separate invocations. - Missing timestamps, transcripts, or correlation evidence are represented as explicit gaps. ## Validation - OpenClaw and PinchBench suites: 88 passed. - Core trajectory, collector, correlation, base-agent, and model suites: 155 passed. - Producer-to-collector projection verifies model calls, token details, tool output, status, and independent timing in `ng_trajectory`. - Ruff, formatting, compilation, shell, and diff checks passed. ## Follow-up pull requests - [#2117 — Add Hermes rollout observations](#2117) - [#2118 — Add Pi rollout observations](#2118) - [#2119 — Add OpenCode rollout observations](#2119) - [#2120 — Add SWE and OpenHands rollout observations](#2120) - [#2122 — Add Harbor rollout observations](#2122) Foundation work is available in [#2114](#2114) and [#2153](#2153), both merged into `main`. --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
## Summary - Add an Equivalence Match verification pattern guide with code examples - Cover exact, normalized, numeric, symbolic, fuzzy, structured, and constraint-based matching - Link relevant resources servers and update the Verification Patterns index ## Testing - `cd fern && npm run check` Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com> Co-authored-by: Seph Mard <seph.mard@gmail.com>
…sponse (#2461) `NeMoGymResponseOutputItem` gained a `BeforeValidator` in #2438 that rejects dicts without a `type` field. `test_run_workflow` mocks the /verify echo with an untagged item, `{"role": "user", "content": "obs"}`, so validating that payload into `AviaryAgentVerifyResponse` now raises. Without the guard, an untagged output-call dict validates as the wrong union member: a web-search call with no `type` becomes `NeMoGymResponseMcpListTools`. The Aviary integration is unaffected. `AviaryAgent.run` posts `verify_request.model_dump()`, and `AviaryResourcesServer.verify` echoes that back, so each item carries its `type`. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
## Summary - Add new `responses_api_models/litellm_model` server for LiteLLM proxy endpoints - `LiteLLMModelServer` extends `openai_model`'s `SimpleModelServer`, overriding only `responses()` to normalize LiteLLM proxy quirks - Normalize `chat.completion` hybrid responses to standard Responses API format (LiteLLM may downgrade `/v1/responses` internally) - Fix `reasoning.effort="none"` (string) to `null` Pydantic validation error for native Responses API responses - `openai_model` remains clean -- no proxy-specific workarounds ## Test - [x] Tested with GPT-5.4 via LiteLLM-backed endpoint -- native response format, reasoning fix applied - [x] Tested with Opus 4.6 via LiteLLM-backed endpoint -- chat.completion hybrid normalized - [x] End-to-end rollout collection verified with both models - [x] 11 unit tests for litellm_model (normalization + server integration) - [x] openai_model existing tests still pass after revert --------- Signed-off-by: Jin Xu <jinx@nvidia.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com> Signed-off-by: Jin Xu <jin.xu999@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Kajal Jain <kajalj@nvidia.com>
## Summary
On OpenSandbox deployments that enforce API-key auth on the server's
`/proxy/*` routes, every proxied execd request fails with `401
{"code":"MISSING_API_KEY"}` **even though `api_key` is configured** —
because the SDK wires the key only into the lifecycle-API clients. The
execd-facing clients (health ping, commands, filesystem) send only
`ConnectionConfig.headers`, which the provider never populated.
The visible symptom is nasty: `Sandbox.create` returns `202`, endpoint
lookups return `200`, but the SDK's internal health wait pings
`/proxy/44772/ping` keyless, gets `401` forever, and creation spins
until `ready_timeout` before failing with
`SandboxReadyTimeoutException`. Exec and file operations through the
proxy fail the same way.
## Change
In `_connection_config`, when `use_server_proxy` is enabled and an
`api_key` is configured, also pass the key as the `OPEN-SANDBOX-API-KEY`
header via `ConnectionConfig.headers` — the SDK merges those headers
into every execd client, so proxied pings/execs authenticate.
Scoped deliberately:
- **Proxy mode only.** In direct-endpoint mode the request terminates at
the sandbox itself, which runs untrusted code and must never see the
key.
- Harmless on servers that don't enforce proxy auth (verified: the
header is accepted either way).
## Testing
Extended `test_connection_config_and_image_policy`:
- proxy mode + api_key → `headers={"OPEN-SANDBOX-API-KEY": ...}`
present;
- direct mode + api_key → no `headers` injected.
Verified against a live enforcing deployment: keyless ping/exec → `401
MISSING_API_KEY`; with the header → `200`, including immediately after
create.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…hands.sh make-build retry (#2446) The retry loop around `make build` in `setup_scripts/openhands.sh` has three bugs: 1. **Wrong exit code captured.** `exit_code=$?` runs after the `if` statement has completed, so it captures the if-statement's status (always 0), not make's. Timeouts are reported as "failed with exit code 0" and exit 124 is never detected. 2. **Infinite loop on final failure.** On the final attempt, a failure neither increments `attempt` nor exits, so the loop re-runs `make build` forever instead of failing the setup. In a Docker build this hangs until the CI job times out; worse, if a late retry "succeeds" against a half-broken tree, the failure is deferred to runtime. 3. **Stale `.venv` between retries.** A first-attempt timeout leaves a partially-built in-project `.venv` behind (`POETRY_VIRTUALENVS_IN_PROJECT=true` is set just above); `make clean` does not remove it. The script has already `cd`'d into `$openhands_dir` at this point, so the removal only targets the OpenHands checkout. This PR captures the exit code in an `else` branch (while it is still make's), `exit`s with it on the final failure, and removes `.venv` before retrying. It also makes the first-attempt timeout env-overridable (`MAKE_BUILD_TIMEOUT_SECONDS`, default unchanged at 2 minutes), so hosts where a healthy `make build` legitimately takes longer can raise it without editing the script — on such hosts the 2-minute guard otherwise kills every healthy first attempt and forces the fallback path above. Signed-off-by: Michal Futrega <mfutrega@nvidia.com>
#2449) The request body's `max_output_tokens` is already mapped into `inference_params` as `tokens_to_generate`, but `get_run_command` never writes it into the generated OpenHands `config.toml` — only `temperature` and `top_p` are forwarded. OpenHands therefore runs with its own default output limit, and a caller-requested cap (e.g. from a training harness that budgets generation lengths) is silently ignored. Forward it as `llm.model.max_output_tokens` when present; behavior is unchanged for requests that don't set it. Signed-off-by: Michal Futrega <mfutrega@nvidia.com>
## Summary Add Hermes production of `ng_agent_observations` for correlated rollouts and update the trajectory capability matrix. Hermes observations use instance-local hooks. Uncorrelated requests retain the existing response path. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 require correlated Gym Model Server capture. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `hermes_agent` | V | V | X | V | V | V | V | C3 remains unavailable because Hermes does not emit standardized per-turn records. ## Captured evidence - root and delegated-agent conversations, including system prompts and plain reasoning - parent and `delegate_task` spawn relationships when directly observed - model-call references from response IDs - model-visible tool outputs, execution status, timestamps, duration, and independent concurrent-call intervals - context-compaction outcomes and available pre/post token estimates - invocation outcomes Existing Hermes callbacks are chained. Observation failures do not mask agent failures. ## Evidence boundaries - Missing Hermes hooks and unattributed child spawns are reported as explicit gaps. - Opaque `reasoning_details` are not normalized. - Compaction summaries and exact adjacent model calls are unavailable; failed compactions do not report a post-compaction token count. - Local terminal execution reports `no_sandbox_runtime`. - External terminal backends report `sandbox_observation_unavailable` because Gym does not receive their lifecycle or resource telemetry. ## Compatibility - observation capture is enabled only for correlated rollouts - the private self-call attachment is removed before verification - verifier payloads, rewards, and token-bearing output retain existing semantics - `ng_agent_observations` is optional and additive ## Validation - Hermes suite: 44 passed - compatibility suite: 239 passed - local E2E: pinned Hermes agent, concurrent tools, model-call correlation, normalized conversation, and trajectory projection - Ruff, formatting, compile, and diff checks passed ## Related pull requests This is **4/7** in the rollout-observability series. Producer PRs 4/7 through 7/7 are independently based on `main`. <!-- stack-links --> - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) - [3/7 #2115 — Add OpenClaw rollout observations](#2115) - [4/7 #2117 — Add Hermes rollout observations](#2117) - [5/7 #2118 — Add Pi rollout observations](#2118) - [6/7 #2119 — Add OpenCode rollout observations](#2119) - [7/7 #2120 — Add SWE and OpenHands rollout observations](#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
Signed-off-by: Jakub Slowikowski <jslowikowski@nvidia.com> Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com> Signed-off-by: ritaneves <rfernandesne@nvidia.com> Co-authored-by: Jakub Slowikowski <jslowikowski@nvidia.com> Co-authored-by: Wojciech Prazuch <wprazuch@nvidia.com> Co-authored-by: Grzegorz Chlebus <gchlebus@nvidia.com>
## Summary Add Pi production of `ng_agent_observations` for correlated rollouts and update the trajectory capability matrix. Pi's JSON event stream is captured while the process runs. Uncorrelated requests retain the existing response path and provider configuration. ## Capability coverage `V` = supported, `O` = partial or path-dependent, `X` = unavailable. C1, C2, and C4 require correlated Gym Model Server capture. | Agent | C1 | C2 | C3 | C4 | C5 | C6 | C7 | | --- | --- | --- | --- | --- | --- | --- | --- | | `pi_agent` | V | V | X | V | V | V | V | C3 remains unavailable because Pi does not emit standardized per-turn records. ## Captured evidence - model-visible system prompt, user instruction, assistant output, tool calls, and tool results - model-call references from response IDs when `model_server` is configured - independent tool-call intervals and outcomes, including overlapping calls - context-compaction outcomes, summaries, token counts, retained boundaries, and adjacent model-call references when available - invocation status from Pi's terminal `agent_end` event - partial evidence from incomplete or timed-out runs ## Evidence boundaries - Pi does not expose subagent hierarchy. - Tool timings use harness receipt timestamps rather than executor timestamps. - Incomplete model-call, tool, and compaction pairs are reported as gaps. - Unknown terminal outcomes remain `unknown` with an `invocation_outcome_unavailable` gap. - The integration runs on the host and reports `no_sandbox_runtime`. ## Compatibility - `model_server` remains optional; direct-provider configuration is unchanged - configured Gym Model Servers use rollout-prefixed routing - response, verification, reward, and token-accounting behavior is unchanged - observation parsing failures do not fail the rollout - the private self-call attachment is removed before verification - `ng_agent_observations` is optional and additive ## Validation - Pi suite: 34 passed - Shared observability, correlation, and collector suites: 82 passed - Prefixed FastAPI response round-trip covered - Ruff, formatting, and diff checks passed ## Related pull requests This is **5/7** in the rollout-observability series. Producer PRs 4/7 through 7/7 are independently based on `main`. <!-- stack-links --> - [1/7 #2114 — Define the rollout observation and correlation contract](#2114) - [2/7 #2153 — Add Claude Code rollout observations](#2153) - [3/7 #2115 — Add OpenClaw rollout observations](#2115) - [4/7 #2117 — Add Hermes rollout observations](#2117) - [5/7 #2118 — Add Pi rollout observations](#2118) - [6/7 #2119 — Add OpenCode rollout observations](#2119) - [7/7 #2120 — Add SWE and OpenHands rollout observations](#2120) --------- Signed-off-by: Michal Bien <mbien@nvidia.com>
## Summary Gym depends on plain `uvicorn`, so `httptools` is not guaranteed to be installed and Uvicorn can silently fall back to its pure-Python h11 parser. This change adds `httptools` as a dependency and explicitly selects it in the Uvicorn configuration. Explicit selection also turns a missing or incompatible wheel into a startup error instead of silently changing the server's parser. ## Performance methodology For testing, added a synthetic benchmark that used a deterministic `SimpleResponsesAPIModel` which returns cached, training-shaped dictionaries. This isolates Gym's dispatch, middleware, serialization, and HTTP serving costs from model generation. This was tested with #2867 The matrix covered: - 32 output tokens (4.9 KB response) at 1,024, 4,096, and 8,192 concurrent requests - 1,024 output tokens (22.6 KB response) at 1,024 and 4,096 concurrent requests - 65,536 output tokens (1.40 MB response) at 1,024 concurrent requests The httptools-only variant improved median throughput by up to 4.2%. The result was neutral at 8,192 concurrency, where server saturation and scheduling variance dominated parser cost. <img width="1205" height="780" alt="Screenshot 2026-08-28 at 5 11 33 PM" src="https://github.com/user-attachments/assets/1d7382de-2ab4-44cf-b198-8d9f5f4aa716" /> <img width="1200" height="233" alt="Screenshot 2026-08-28 at 5 11 42 PM" src="https://github.com/user-attachments/assets/8606eea7-6baa-46c9-8255-d1a62ee813da" /> ## Test plan - [x] `uv run pytest tests/unit_tests/test_server_utils.py -q` - [x] `uv run pre-commit run --files nemo_gym/server_utils.py pyproject.toml uv.lock` --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
## What does this PR do? Clarifies how to interpret the trajectory capability matrix for the 0.6.0 documentation: - explains that the matrix reports observed producer coverage rather than absolute schema capability - adds an upfront Key caveats summary for path-dependent and conditional behavior - makes the relationship between model-call capture, trajectories, and direct-provider calls explicit - adds practical guidance for interpreting V, O, and X - restructures producer-specific coverage notes into scannable bullets ## Validation - make docs-check — 0 errors; the authentication-only redirects check is deferred to CI - python3 tests/unit_tests/test_fern_docs_links.py — 14 tests passed - uvx pre-commit run --all-files — passed ## Checklist - [x] I have read the contributing guidelines. - [x] The change is focused; unrelated drive-by edits are tracked separately. - [x] Tests are N/A for this docs-only change; documentation validation passes locally. - [x] Pre-commit checks pass locally. - [x] All commits have DCO sign-off. Signed-off-by: Michal Bien <mbien@nvidia.com>
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? Adds `gym compare`, a new CLI command that diffs a baseline eval run against a candidate run and writes a report. Related epic: #1493 and issue addressed by this PR: #2531 ### Note This to be fully completed needs statistical testing and [rollout quality](https://rfc.frontier-evals.nvidia.com/r/gym-rollout-verification#ann=0da3dd8330c7), this PR add a skeleton of the report and machine readible output file, report deltas and ci. Later output of statistical tests and rollout quality work can be just plugged in follow-up steps. ### Output report example [compare_report.md](https://github.com/user-attachments/files/31390538/compare_report.md) ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>
…ine dispatch (#2892) ## Summary The openai-2.44 pin rework (65129dd, #2456) stopped popping `strict` from function tool definitions in `responses_to_chat_completion_create_params`. This restores the pre-#2456 strip. `strict` is an OpenAI-hosted enforcement flag with no Chat Completions semantics on self-hosted engines: vLLM 0.25.1's `FunctionDefinition` keeps it, and chat templates that render unknown function keys (e.g. Nemotron's `render_extra_keys`) inject `<strict>True</strict>` into every `<function>` block. That perturbs the prompt of every tool-bearing request off-distribution. ## Observed impact In RL training (jobs 6707880/6712632 vs baseline 6682989): ~17% of sequences per step ran away to the max-token cap, collected reward-0 truncation penalties, and were masked by sequence logprob error (1e4–1e14). Restoring the strip returns the engine request to byte-parity with the pre-#2456 tree. ## Question for reviewers Was dropping the strip in #2456 intentional? If `strict` support on self-hosted engines is planned, a config flag may be preferable — but unconditional stripping matches pre-#2456 behavior and `strict` has no Chat Completions semantics today. ## Changes - `nemo_gym/responses_converter.py`: pop `strict` from function tool dicts before building `NeMoGymChatCompletionToolParam` - `tests/unit_tests/test_responses_converter.py`: assert `strict` does not reach the engine params 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Pranav Thombre <pthombre@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fixes four inaccuracies in the Switchyard model-server docs page, found by cross-checking every claim against the Gym and Switchyard 0.2.0 source: `--model` resolves a route's `id` field rather than its TOML table name, the durable routing log is unavailable to the Gym-hosted proxy (not standalone-binary-only in Switchyard), the stats snapshot counts per-target `calls` not requests, and the switchyard-server build line needed `--locked -p switchyard-server`. Filed against nvbug 6674632 for the 0.6.0 release. `fern check` passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Lin Jia <linj@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## What does this PR do? Adds SWE-bench Pro as a standalone resources server using NeMo Gym's `AsyncSandbox` API. SWE-bench Pro is kept separate from the existing SWE-bench server because its public evaluator is a standalone Modal/local-Docker script with different task metadata, images, run scripts, parser scripts, and grading flow; it does not expose the `swebench` package's `run_instance` seam. ### Upstream evaluator parity The verifier is ported from the pinned [`swe_bench_pro_eval.py`](https://github.com/scaleapi/SWE-bench_Pro-os/blob/ca10a60a5fcae51e6948ffe1485d4153d421e6c5/swe_bench_pro_eval.py). Relevant upstream functions retain their names and structure so they can be compared directly: - `strip_binary_hunks` is copied directly (apart from repository formatting). - `create_entryscript` preserves the upstream reset, patch, setup, test, and parser sequence. - `assemble_workspace_files` preserves the upstream workspace contract. - `grade_output` extracts the pass/fail logic from upstream `main()`. Substantive adaptations inside those functions are marked `NeMo Gym change`. The intentional differences are: - parse dataset list fields with `json.loads`/`ast.literal_eval` instead of `eval`; - read Dockerfiles, `run_script.sh`, and `parser.py` from prepared JSONL rows instead of a runtime upstream checkout; - retain patch-application status for the structured verifier response; and - replace Modal/Docker orchestration with a separate NeMo-specific `run_verification` function. ### Sandbox and patch flow `seed_session` creates the agent workspace from the task's immutable image digest. At verification time the server extracts committed, modified, and untracked changes relative to `base_commit`, stops the agent sandbox, and uploads the resulting patch plus evaluator assets into a fresh verification sandbox. Golden-patch mode follows the same verification path but uses the dataset patch, allowing evaluator and image health to be tested independently of an agent. The server also: - stops verification sandboxes in `finally` blocks and cleans abandoned session sandboxes during application shutdown; - persists the candidate patch, generated entry script, parser output, test stdout/stderr, execution output, and patch status per request; and - reports extraction, sandbox, patch-application, parser, and test failures as structured zero-reward responses. ### Reproducible data and configuration The preparation script pins both the Hugging Face dataset revision and upstream evaluator commit, embeds each task's evaluator assets, and resolves Docker Hub tags to immutable image digests. This avoids runtime dependency on the upstream repository layout and avoids case-sensitive tag rewriting by registry mirrors. The primary resources-server config remains in normal model-patch mode. A separate example-only server uses golden-patch mode with the simple agent so CI can validate all five checked-in examples without depending on a particular coding-agent harness. The OpenCode harness also falls back to Python when an image lacks `curl` and strips the newline from `pwd` before downloading its exported trajectory. ### Validation - 17 SWE-bench Pro unit tests pass. - Example data and all five checked-in rollouts pass `gym env test ... +should_validate_data=true`. - The five example golden-patch rollouts completed with rewards `[1.0, 1.0, 1.0, 0.0, 1.0]`; the unresolved NodeBB example applied its patch and completed evaluation but did not report every required test as passed. - The OpenCode agent tests pass (3 tests). - Ruff and scoped pre-commit checks pass. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). - [x] Validation runs with opencode setup: https://wandb.ai/nvidia/sdevare-gym-eval/runs/iugntgc0?nw=nwusersugamdevare https://wandb.ai/nvidia/ffrujeri-gym-eval/runs/pz5753j6/logs --------- Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
…2886) ## Summary - pass the request context into workplace assistant verification so the active session can be identified - remove the per-session tool environment after verification, including when scoring raises - cover successful and failed verification cleanup ## Test plan - [x] `uv run gym env test --resources-server workplace_assistant` (32 passed) - [x] `pre-commit run --files resources_servers/workplace_assistant/app.py resources_servers/workplace_assistant/tests/test_app.py` Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Wraps [ViBench](https://github.com/ViBench/vibench-public) as a NeMo Gym environment: a model builds a web app from a PRD, and grading stands the app up for real, seeds it through its own UI, and drives it in a browser against a human-written test plan. The score reflects what a user can reach, not what the diff contains. Adds `resources_servers/vibench` and `responses_api_agents/vibench_agent`. ## Design The agent owns the build sandbox and copies the finished app out as a tarball; the resources server unpacks it and grades it in a fresh compose project, one per test plan, via ViBench's `run-seed.py` → `run-evaluate-post-seeding.py`. The sandbox is never shared. Reaching into the agent's box needs `serialize()`/`connect()`, which only the OpenSandbox provider implements, so the shape `swebench` uses cannot run on Docker, Apptainer or enroot. See [#2082](#2082) for the design discussion; this PR has no dependency on it or on #2085. `vibench_agent` subclasses `OpenCodeSandboxedAgent` and overrides only sandbox acquisition and harvesting. ## Reward One row is one `(app, artifact)` pair: built once, graded across that artifact's test plans. Reward is the mean normalized score, continuous rather than binary, with per-plan values in `reward_components`. `build_failed`, `seeding_failure_rate` and `plans_graded_rate` are reported separately, because a mean reward cannot distinguish a weak app from one that never built or could not be seeded. `REVERIFY_MODE` is `UNSUPPORTED` — grading depends on live app and database state. ## Validation Five example tasks graded end to end on a Docker host. Scores spread from 0.00 to 1.00 with partial credit in between, and every zero carried a real scorecard with per-step diagnoses (HTTP 500s, SQL errors) rather than a harness failure. The reward tracks capability: a weaker policy model scored 0.0 on a task with 0/19 steps, its app unreachable, where a stronger model scored 1.0 on the same task with the same graders. Scores also vary across repeats of the same model, since it does not build the same app twice. Quantifying that — and separating it from any variance in the LLM-driven verifier — is what reward profiling is for, so `verified: false` stands. `data/example_rollouts.jsonl` holds a five-task run. ## Reviewer notes - **`network: host` is deliberately not used.** The in-sandbox harness reaches the policy model at `get_server_url(...)` → `127.0.0.1:<port>`, which inside a bridged container is the container itself; the harness then makes zero LLM calls and exports an empty app with nothing logged. `configs/docker.yaml` instead adds `host.docker.internal` via `host-gateway` and the agent rewrites loopback model URLs to it. Only `policy_model` binds `0.0.0.0`; the servers holding grader credentials stay on loopback. Residual: the sandbox can reach host-published TCP ports, which inference requires. - **Grading timeouts interrupt rather than terminate.** ViBench tears its compose project down in a `finally`, which CPython does not unwind on SIGTERM. Escalation is SIGINT → SIGTERM → SIGKILL. Verified with a forced timeout on a real host: SIGTERM leaked two containers and a network, SIGINT cleaned up fully. - **`artifact_dir` is a filesystem path shared by agent and resources server.** Not a new constraint — grading already shells into a local Docker daemon. - **Requires `docker-compose` on `PATH`.** ViBench's scripts use the legacy name; Docker 29.x ships only the plugin. The failure misleads, so it is called out in the README. - **Task fields are top-level, not under `verifier_metadata`**, following `swebench` since `seed_session` and `verify` share one typed request model. Happy to move them. - **Grading runs on the resources server's Docker daemon**, not in a Gym sandbox, because ViBench's grading stack is multi-container. Single-image packaging is the prerequisite for multi-host. - **`mvp` artifacts only.** ViBench ships reference implementations for 21 of 24 apps, so `feature-ri` is a follow-up; it needs a starting codebase staged into the sandbox the way the PRD already is. - **Row size**: rollout rows carry the harness's own stdout/stderr, so `example_rollouts.jsonl` is ~500 KB for five tasks. Happy to trim. ## Security The tarball comes from a box the model controlled, so it is untrusted: unpacking refuses members and links resolving outside the app dir and passes `filter="data"`, and artifact paths outside `artifact_dir` are refused and never deleted. Grader credentials are scrubbed from captured output before it is stored, since that output ships in the rollout JSONL. Test plans and `test_assets/` are never staged into the build box. ## Testing `gym env test --resources-server vibench` passes. 75 unit tests (60 resources server, 15 agent), coverage 99%. `pytest tests/unit_tests/` 2621 pass. `pre-commit run --all-files` clean. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally. - [x] Pre-commit checks pass locally (`pre-commit run --all-files`). - [x] All commits have DCO sign-off (`git commit -s`). --------- Signed-off-by: raghavendran ramakrishnan <raramakrishn@nvidia.com> Co-authored-by: raghavendran ramakrishnan <raramakrishn@nvidia.com>
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? 1. Enable model-specific params for sbatch_external_vllm.sh: benchmarks/nemotron_3.5_super/vllm_configs/*, benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh 2. Pass default overrides: benchmarks/nemotron_3.5_super/sandbox_utils.yaml, benchmarks/nemotron_3.5_super/sbatch_external_vllm.sh, benchmarks/swebench/pro/opencode.yaml 3. Fix GlobalConfig struct: nemo_gym/global_config.py, tests/unit_tests/test_global_config.py 4. Rollout collection always prints 500 error info: nemo_gym/rollout_collection.py 5. Misc cleanup: responses_api_agents/opencode_sandboxed_agent/app.py ## Checklist - [ ] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [ ] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [ ] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [ ] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [ ] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Brian Yu <bxyu@nvidia.com>
<!-- Thanks for contributing to NeMo Gym! Please fill out the sections below. --> ## What does this PR do? 1. Terminus 2 initial implementation. Functionally works, but not fully stable: responses_api_agents/terminus_2_sandboxed_agent, benchmarks/terminal_bench_2_1/terminus_2.yaml 2. Append to Super 3.5 evals config: benchmarks/nemotron_3.5_super/eval_container_config_with_staged.yaml, benchmarks/nemotron_3.5_super/eval_container_config.yaml ## Checklist - [ ] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [ ] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [ ] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [ ] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [ ] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Brian Yu <bxyu@nvidia.com>
`vibench_agent` shipped without a `README.md` in #2674. `ng_test_all` filters testable components to directories containing one, then asserts the filtered count matches the discovered count — so discovery found 174 components and tested 173, failing that assertion in all eight server-suite shards even though every selected module test passed: ``` AssertionError: Mismatch on the number of total modules found (174) and the number of actual modules tested (173)! Extra candidate paths: - responses_api_agents/vibench_agent ``` `nemo_gym/cli/env.py:1114`, on `main` at a2c5222. This adds the missing README. Docs-only — no code or config changes. ## Contents - The build → harvest → verify flow, and that everything about driving OpenCode is inherited from `opencode_sandboxed_agent` (only sandbox acquisition and harvesting are overridden). - Why the agent owns its sandbox instead of sharing one: reaching into the agent's box needs `serialize()`/`connect()`, which only OpenSandbox implements, so the `swebench` shape cannot run on Docker, Apptainer or enroot. See #2082. - Why `configs/docker.yaml` must be used instead of the stock provider config — the stock 180s exec timeout kills long installs, and the harness is told the model is at `http://127.0.0.1:<port>`, which inside a bridged container is the container itself, so it makes zero LLM calls and exports an empty app. - An explicit warning not to "fix" that with `network: host`, and the `0.0.0.0` exposure caveat for the policy-model bind. Setup, task-row generation, reward definition and grading are not duplicated here; they stay in the [resources server README](https://github.com/NVIDIA-NeMo/Gym/blob/main/resources_servers/vibench/README.md). ## Note on the other CI failure The `Test Wheel Use` failure in that run is independent of this change: the wheel test enables prereleases/`unsafe-best-match`, unbounded `mlflow>=3.15.1` resolved to 3.15.2, which selected `sqlalchemy 2.1.0rc1`, whose source metadata uv rejects (`duplicate normalized extra name 'mssql-pymssql'`), aborting `arc_agi_resources_server` startup. That needs a SQLAlchemy pin below 2.1 or prerelease containment in transitive resolution — happy to open a separate issue or PR for it if useful. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: raghavendran ramakrishnan <raramakrishn@nvidia.com> Co-authored-by: raghavendran ramakrishnan <raramakrishn@nvidia.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Fixes #2699 and another bug found while testing --------- Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Fixes #2221. Passing a config without fully defined mode emits a warning. If user passes `--strict` - the error is raised for such cases. --------- Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
Contributor
|
🌿 Preview your docs: https://nvidia-preview-spanev-gdpval-devel-main-sync.docs.buildwithfern.com/nemo/gym |
…e strict chat schema Signed-off-by: Serge Panev <spanev@nvidia.com>
… path-only fingerprint drift Signed-off-by: Serge Panev <spanev@nvidia.com>
… schema Signed-off-by: Serge Panev <spanev@nvidia.com>
…s for regime parity Signed-off-by: Serge Panev <spanev@nvidia.com>
Kh4L
marked this pull request as ready for review
September 3, 2026 17:12
github-actions
Bot
added
the
sla:triage-overdue
This was referenced Sep 4, 2026
This was referenced Oct 22, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges
main(272 commits, headcd6acf792) intogdpval-devel.gdpval-develstayed frozen at its hardware-validated head (2ce39b13e) until this branch passed the same real-checkpoint validation ladder; it has now passed 3/3 (offline refit + resume compatibility; 5-task real-checkpoint rollout smoke with 654 policy calls and 0×422; TRUE3 judging 80/80 trials with fail-closed resume).Merge notes
rollout_collection.pyunifies devel's dispatch budget/drain/latency tracker with main's opt-in sidecar failure routing, loud drop banners, per-rollout latency stamping, and exporter uploads; two textual-merge bugs fixed (timeimport shadowing, lostresult_strsinit). Venv-path helper renamed to main'sget_venv_path.local_vllm_modeltakes main'svllm==0.25.1+ flashinfer pin.Regressions found by hardware validation and fixed here
extra="forbid"onNeMoGymChatCompletionCreateParamsNonStreamingcascaded into the message TypedDicts and rejected payloads Gym's own producers emit and the previous ingress silently dropped:chat_template_kwargs,thinking,output_config(strictness retained; typos still 422).{role, content, tool_calls[{id,type,function}]}/{role, content, tool_call_id}), verified byte-identical against the pinned baseline over a 67-shape matrix; history objects untouched.input_audio.formattoken (the SDK literal is wav/mp3-only; Gym emits m4a/flac/ogg/aac/aiff).run_e2e_multistagenow forwardsroute_failures_to_sidecar(it silently reverted to raising).endpoint_*liveness knobs as connection fields, so relocating a run or adding endpoint knobs no longer stales journals.chat_template_kwargsis accepted but forwarded only behindforward_request_chat_template_kwargs(default off) so agents attachingenable_thinkingon every call do not silently change the generation regime relative to validated campaigns.Compatibility notes
.stale.*, never deleted) because result-relevant config keys were added on main; land syncs between campaigns.route_failures_to_sidecardefaults toFalseon main (a failed/runraises); GDPVal configs that relied on sidecar routing must set it explicitly._ng_rollout_latency_ms.Validation
mainin this environment); 644 GDPVal + Stirrup tests; 550 core/proxy tests.1342.0334; job68499265/5 rollouts, 654 calls, 0×422; jobs6793203/679541980/80 TRUE3 trials with byte-identical zero-POST resume.