Repository navigation
Add Daytona sandbox provider - #1
Closed
hemildesai wants to merge 149 commits into
Closed
hemildesai wants to merge 149 commits into
hemildesai wants to merge 149 commits into
Conversation
hemildesai
force-pushed
the
hemil/daytona-sandbox
branch
2 times, most recently
from
June 9, 2026 16:48
13dd946 to
4ba2396
Compare
hemildesai
pushed a commit
that referenced
this pull request
Jun 9, 2026
…uting (NVIDIA-NeMo#1367) ## Summary Two coupled changes that together fix walltime-induced rollout loss on multi-node GDPVal runs. **1. Per-task timeout** — default 3h30m, env `STIRRUP_PER_TASK_TIMEOUT_S` overrides. Wraps `await future` in `asyncio.wait_for`; on timeout, `ray.cancel(future, force=True)` + raises `TaskPerAttemptTimeoutError`. Logs once per process at first dispatch. **2. Failure classification + sidecar routing** — at the two `_build_failed_run_payload` callsites, every failure is classified into one of five classes and persisted accordingly: | class | persist | retry on chain-hop 2 | |---|---|---| | `kill_shaped` (Ray actor died, SIGTERM, OOM, node failure) | NO row anywhere | yes, unbounded (per-attempt timeout bounds wallclock) | | `timeout_exceeded` | 1 sidecar entry, `_ng_failure_terminal=True` | no | | `skipped` (TaskSampleSkipError) | 1 sidecar entry, terminal | no | | `transient` (verify-side 5xx, ConnectionError, asyncio.TimeoutError) | sidecar entry per attempt | yes, up to `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` (default 3) | | `legitimate` (real Python exception with user code in traceback) | sidecar entry per attempt | yes, up to max_attempts | Successes still write to `<output_jsonl_fpath>`; failures write to `<stem>_failures.jsonl`. `_load_from_cache` reads both: main jsonl is the success ledger, sidecar tracks attempts + terminal flags. The retry set is `materialized_inputs − (successes ∪ terminal ∪ maxed_out)`. ## Why this matters Without #1, a single pathological task that exceeds Slurm walltime can permanently consume every chain-hop's compute and never complete. Without #2, walltime-killed in-flight rollouts permanently disappear on chain-hop 2. The old `_load_from_cache` dedup keyed on `(task_index, rollout_index)` regardless of `-failed` status, so synthetic `-failed` rows written during the SIGTERM grace window looked already-done to the resumer. Worse, under harsh kills (SIGKILL, OOM) the `-failed` row write itself is non-atomic — we couldn't depend on it being there to filter. Using "row absent from main jsonl" as the canonical "needs retry" signal sidesteps both problems: kill_shaped writes nothing at all, so the disk state survives arbitrary kill timing. See the debug write-up for the production incident and design rationale: https://gitlab-master.nvidia.com/agronskiy/idea/-/blob/main/reports/debug/20260519T1011-gdpval-missing-histories.md ## Kill-shaped detection `_classify_rollout_failure` uses Ray's actor-died classes (`RayActorError`, `WorkerCrashedError`, `NodeDiedError`, `OutOfMemoryError`, `LocalRayletDiedError`) plus a user-code-in-traceback fallback for `RayTaskError`. The fallback distinguishes a real user exception (frames under \`responses_api_agents/\` or \`stirrup/\`) from Ray's internal post-mortem (e.g. summary-builder hitting a vanished worker log after Slurm's epilogue scrubbed `/tmp/ray`) — the latter is the walltime / SIGTERM signature and routes to `kill_shaped`. Detection fails open: if Ray's exception surface drifts, everything goes to bounded-retry `legitimate` instead of unbounded-retry `kill_shaped` — safe, not catastrophic. ## Knobs - `STIRRUP_PER_TASK_TIMEOUT_S` — per-attempt timeout (default 12600 s = 3h30m). - `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` — max retries per `(task_index, rollout_index)` (default 3). ## Test plan - \`python -m py_compile\` clean for both edited files. - ruff format clean. - Suggested smoke: run a GDPVal eval with `STIRRUP_PER_TASK_TIMEOUT_S=60`, confirm the log line is emitted once and that long-running rollouts get cancelled cleanly with `_ng_failure_class=timeout_exceeded` in `<stem>_failures.jsonl`. - Suggested integration: deliberately kill the deployment srun mid-run (`scancel -s TERM`), verify `_failures.jsonl` is unchanged (kill_shaped writes nothing) and that on chain-hop 2 the killed rollouts re-dispatch. --------- Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
hemildesai
force-pushed
the
hemil/sandbox-api-part-1
branch
3 times, most recently
from
June 11, 2026 21:36
8127481 to
428238f
Compare
hemildesai
force-pushed
the
hemil/sandbox-api-part-1
branch
3 times, most recently
from
June 22, 2026 18:47
f455bac to
ab53503
Compare
ananthsub
force-pushed
the
hemil/sandbox-api-part-1
branch
from
June 23, 2026 00:24
ab53503 to
3706a55
Compare
## Summary Parent PR: NVIDIA-NeMo#1368 Refs NVIDIA-NeMo#1337 This is the first smaller PR split out from NVIDIA-NeMo#1368. It keeps the scope to the provider-neutral sandbox API, the OpenSandbox provider, and the Mini SWE Agent 2 evaluation integration. Observability is intentionally left out for a follow-up PR. ### Features - Adds the public `nemo_gym.sandbox` facade with async and sync sandbox clients, provider registration, image rewrite support, sandbox specs/handles, and batch create support. - Adds the OpenSandbox provider with create/connect/exec/file/close operations, SDK pool-backed batch creation, retry handling, create probes, direct exec endpoint support, and nested provider configuration sections. - Adds `responses_api_agents/mini_swe_agent_2`, a sandbox-backed mini-swe-agent v2 integration for SWE-bench style evals, including sandbox resource profiles, task metadata propagation, reward aggregation, and `ng_collect_rollouts` usage docs. - Adds focused unit coverage for the sandbox facade, provider registry, OpenSandbox provider behavior, Mini SWE Agent 2 run/aggregation behavior, and sandbox environment adapter. - Moves sandbox-related dependencies behind the `nemo-gym[sandbox]` optional extra. ### Notes - This PR does not include the sandbox observability module from NVIDIA-NeMo#1368. - The Mini SWE Agent 2 README avoids internal deployment names and user-specific paths; examples use placeholders and local `data/` / `results/` paths. ## Validation Completed on the squashed commit: ```bash uv run ruff check nemo_gym/sandbox responses_api_agents/mini_swe_agent_2 tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py ``` Result: `All checks passed!` ```bash uv run pytest tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py responses_api_agents/mini_swe_agent_2/tests/test_app.py responses_api_agents/mini_swe_agent_2/tests/test_sandbox_environment.py -q ``` Result: `44 passed, 2 warnings` ```bash uv run coverage run --source=nemo_gym.sandbox,responses_api_agents.mini_swe_agent_2 -m pytest tests/unit_tests/test_sandbox.py tests/unit_tests/test_opensandbox_provider.py responses_api_agents/mini_swe_agent_2/tests/test_app.py responses_api_agents/mini_swe_agent_2/tests/test_sandbox_environment.py -q uv run coverage combine results uv run coverage report --include='nemo_gym/sandbox/*,responses_api_agents/mini_swe_agent_2/*' --fail-under=90 ``` Result: focused coverage `92%`. Kubernetes smoke validation: - Model: `Qwen/Qwen3.5-27B`, served by SGLang with DFLASH draft model `z-lab/Qwen3.5-27B-DFlash`. - Launched the Mini SWE Agent 2 stack through the documented `ng_collect_rollouts` path. - Ran 8 SWE-bench Verified samples with 8 repeats and 64-way rollout concurrency against the OpenSandbox internal service path. - Result: 64/64 rollout rows, `pass@8=0.875`, 7/8 tasks resolved, mean reward `0.765625`, eval error rate `0.0`, reward profile completion `100%`. - Cleanup completed with no leftover sandboxes for the successful internal-service run. Full SWE-bench Verified validation: - Model: `Qwen/Qwen3.5-27B`, served by SGLang with DFLASH draft model `z-lab/Qwen3.5-27B-DFlash`. - Ran 500 SWE-bench Verified samples with pass@1, 500-way rollout concurrency, `step_limit=250`, and OpenSandbox cleanup metadata. - Result: 500/500 rollout rows, `pass@1=0.698`, 349/500 tasks resolved, mean reward `0.698`, eval error rate `0.6`, tests status rate `99.0`, reward profile completion `100%`. - Job duration: `4h6m`; rollout collection duration: `4h04m`. - Cleanup left no sandboxes with the run labels `run_family=mini-swe2-firstpr-q35-cell-full-p1-r9` or `cleanup_id=full-p1-r9-single-20260521-053410`. --------- Signed-off-by: Hemil Desai <hemild@nvidia.com>
moves abstention environment to environments/ --------- Signed-off-by: Christian Munley <cmunley@nvidia.com>
migrates reasoning gym simple agent environment to environments/ did not delete anything from resources_server yet (such as config or example rollouts, i suppose its safer to do in a followup PR?) --------- Signed-off-by: Christian Munley <cmunley@nvidia.com>
hemildesai
force-pushed
the
hemil/daytona-sandbox
branch
from
June 23, 2026 03:59
fcc8418 to
e703f21
Compare
…mediation (NVIDIA-NeMo#1657) ## Summary - Bump `mlflow`/`mlflow-skinny` `>=3.12.0` → `>=3.14.0` to fix CVEs - Add `grpcio>=1.81.0rc1` as a `uv` `constraint-dependency` (transitive via ray) to fix CVEs - Bump `torch` `2.5.0` → `2.12.0` and `torchvision` `0.20.0` → `0.27.0` in `longmt_eval` via [kajalj22/SEGALE@a7c9a1a](kajalj22/SEGALE@a7c9a1a) which relaxes the upstream `torch==2.5.0` pin 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Kajal Jain <kajalj@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## What Adds a **GitHub** link to the docs navbar via Fern's [`navbar-links`](https://buildwithfern.com/learn/docs/configuration/site-level-settings) in `fern/docs.yml`: ```yaml navbar-links: - type: github value: https://github.com/NVIDIA-NeMo/Gym ``` ## Why Per NVIDIA-NeMo#1428, the published docs (`docs.nvidia.com/nemo/gym`) don't surface a link to the source repository, making it harder for users to find the code, report issues, or contribute. With this change the repo is reachable from every docs page via the standard GitHub icon in the navbar. ## Notes - Uses Fern's first-class `github` navbar-link type (`value` holds the repo URL, per Fern's site-level settings docs). This is independent of the `nvidia` global theme, which only controls the logo/right-text. - Docs-config only; no code or content pages changed. YAML validated locally. Closes NVIDIA-NeMo#1428 Signed-off-by: Abhay <abhaysingh0293@gmail.com> Co-authored-by: Lawrence Lane <llane@nvidia.com>
Vendors the [github/gh-stack](https://github.com/github/gh-stack) agent skill into `.claude/skills/gh-stack/` so coding agents working in this repo know how to drive stacked PRs (create, push, rebase, sync, navigate) non-interactively. NeMo Gym now uses [GitHub Stacked PRs](https://github.github.com/gh-stack/) for larger changes. Installed via `gh skill install github/gh-stack gh-stack`. Bottom of a 2-PR stack; the contributor guide that references this skill follows in NVIDIA-NeMo#1617. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…eMo#642) Clean up subprocess for the session id at the end of /verify --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
move arc agi environment to environments/ --------- Signed-off-by: Christian Munley <cmunley@nvidia.com>
## Summary - Bumps the pinned vLLM version in `responses_api_models/local_vllm_model` (and its GenRM subclass `responses_api_models/genrm_model`) from `0.17.0` → `0.20.0`. This matches the matching the version [NeMo-RL `main` pins in its `[vllm]` extra](https://github.com/NVIDIA-NeMo/RL/blob/main/pyproject.toml). This keeps the in-process vLLM engine Gym spins up for local eval/rollout on the same engine NeMo-RL trains and evaluates against. - Mac pin left at `0.11.0` (latest cleanly pip-installable on macOS; the local vLLM model realistically only runs on Linux GPU hosts). - Updates a stale `0.17.0`-version comment in `app.py` around the `enable_return_routed_experts` → `return_routed_experts` alias (verified the arg still exists in 0.20.0). ## Test plan Validated in a `vllm/vllm-openai:v0.20.x` container with this worktree mounted - [x] `local_vllm_model` unit tests pass (incl. `_configure_vllm_serve` / arg-parser path) - [x] `ng_run` builds the isolated per-server venv installing **exactly `vllm==0.20.0`** and reaches "servers ready" - [x] **DP1/TP1** dense (`Qwen3-0.6B`): spinup + end-to-end chat completion - [x] **TP=2 + MoE** (`Qwen3-30B-A3B`): weights sharded across both GPUs, server up, inference works - [x] **DP=2** dense (`Qwen3-0.6B`): two `EngineCoreActor` ranks (one per GPU) via the `create_dp_placement_groups` loop + per-rank `_init_data_parallel`, inference works --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
## Summary - add OpenSandbox provider tests for extension validation, float resource normalization, and retry-loop guard branches - add public sandbox/registry tests for missing-spec status, builtin provider loading, and sync async-operation timeouts - raises focused sandbox coverage from 98.34% to 99.86%, leaving only an unreachable defensive OpenSandbox create fallback uncovered ## Validation - `UV_CACHE_DIR=/tmp/uv-cache uv run pre-commit run --files tests/unit_tests/test_opensandbox_provider.py tests/unit_tests/test_sandbox.py` - `UV_CACHE_DIR=/tmp/uv-cache uv run pytest tests/unit_tests/test_opensandbox_provider.py tests/unit_tests/test_sandbox.py --cov=nemo_gym.sandbox.providers.opensandbox --cov=nemo_gym.sandbox.providers.registry --cov=nemo_gym.sandbox.api --cov-report=term-missing` - `UV_CACHE_DIR=/tmp/uv-cache uv run pytest tests/unit_tests/ -x` Signed-off-by: Hemil Desai <hemild@nvidia.com>
current example rollouts are from a older version which have a bug not including the reset observation in the output Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>
moves circle_click and circle_count to environments/ --------- Signed-off-by: Christian Munley <cmunley@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>
move calendar environments to environments/ --------- Signed-off-by: Christian Munley <cmunley@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>
moves code_gen to environments/ --------- Signed-off-by: Christian Munley <cmunley@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>
…live desync errors (NVIDIA-NeMo#1555) uvicorn closes idle keep-alive connections after 5s by default ([ref](https://uvicorn.dev/settings/#timeouts)) while the aiohttp client keeps them pooled for 15s by default ([ref](https://docs.aiohttp.org/en/stable/client_reference.html#connectors)), causing ServerDisconnectedError when the client reuses a socket the server already closed. This PR set the client TCPConnector `keepalive_timeout` to 15s and the uvicorn `timeout_keep_alive` to 30s so pooled sockets are guaranteed live when reused. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
move ether0 environment to environments/ Signed-off-by: Christian Munley <cmunley@nvidia.com>
…port (NVIDIA-NeMo#1663) Adds multi-reference model support and Bradley-Terry MLE ELO computation with multiple reference models in order to directly match AA's ELO score calcualtion algorithm: [GDPval-AA v2](https://artificialanalysis.ai/methodology/intelligence-benchmarking#gdpval-aa) > After collecting pairwise rankings, we fit them to a Bradley-Terry model via maximum likelihood estimation and compute confidence intervals using the sandwich estimator to establish our final Elo metric This is one of many changes needed to match AA's v2 benchmark. --------- Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com> Co-authored-by: Virginia Adams <vadams@oci-hsg-cs-001-vscode-02.cm.cluster> Co-authored-by: Alex Gronskiy <agronskiy@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
hemildesai
force-pushed
the
hemil/daytona-sandbox
branch
from
July 6, 2026 16:28
9e0bd89 to
ca552e8
Compare
Signed-off-by: Hemil Desai <hemild@nvidia.com>
## Summary
Make `resume_from_cache` work for multi-stage GDPval/stirrup ELO runs.
Previously the multistage driver bypassed the standard
rollout-collection resume: a rerun re-planned and re-executed every
stage, a crash lost in-progress results, and with `multistage.seed`
unset the adaptive task/reference sampling diverged on rerun.
## What changed
- **Fingerprinted stage journal** (`<stem>_multistage_state.jsonl`):
per-stage `planned` record (chosen references, task ids, seed, prior
ELO) written *before dispatch* + a `complete` marker at stage end. A
config fingerprint invalidates stale caches.
- **Stage-keyed instance of the core success/failure-sidecar contract**:
successes → main jsonl, non-kill failures → attempt-capped
`_failures.jsonl`, kill-shaped → nowhere. Failures never contaminate the
main jsonl or ELO.
- **Resume-aware `run_multistage_stages`** (pure, injectable
`StageResume` seam): completed stages reuse cached judging results and
re-fit ELO from rows; an interrupted stage replays its recorded
references (deterministic even when `seed` is `None`) and re-dispatches
only not-yet-done `(stage, task, rollout)` rows. Per-task reuse within a
partially-done stage is served by the stirrup agent's
reference-subset-keyed `/verify` cache (`rerun_incomplete`).
- Fixes a fresh-run persistence gap: `_prepare_resume` now always
returns a writing `StageResume`, so a fresh run persists the journal +
rows (previously nothing was written on the first run, making resume
impossible).
## Test plan
- **Unit:** 63 tests pass in `resources_servers/gdpval/tests/` — incl.
fresh→resume cycle, interrupted-stage partial re-dispatch, plan replay
with `seed=None`, failure routing (attempt cap / terminal), and
fingerprint invalidation.
- **End-to-end on HSG** (nano-3.5, gdpval multistage comparison, stages
`[5, 5×num_models:2]`):
1. Launched a run; let stage 1 complete and stage 2 begin (4/5 stage-2
deliverables produced).
2. Interrupted the job mid–stage 2 (`nel kill`).
3. Ran `nel resume` → run completed to **SUCCESS**.
4. Verified from the run logs that resume worked as intended:
- `resuming multi-stage run from cache (fingerprint match)`
- `stage 1/2 reused from cache: eval ELO = 720.0 (5 cached rollout(s))`
— stage 1 not re-run
- `stage 2/2 (replayed from journal): 5 task(s) vs 2 ref(s)
['gemma4_26b','gptoss_120b']` — references replayed from the journal
(with `seed=null`, only possible via persistence)
- stage 2 reused the 4 pre-interrupt deliverables + computed the 1
remaining; final rollouts = 5 stage-0 + 5 stage-1.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
…eMo#1902) Fixes NVIDIA-NeMo#1899 and disambiguates `finance_sec_search` variants --------- Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
torch version bump caused model loading errors vecalign 2.0.0 was also released which changed dependency structure Signed-off-by: Jeff Willette <jwillette@nvidia.com>
…y worker (NVIDIA-NeMo#1937) ## Problem SciCode's `verify()` dispatches each sub-step to a `ray.remote` worker that runs `subprocess.run([sys.executable, "-c", program])`. The worker's `sys.executable` is whatever Python the Ray cluster runs — which is not guaranteed to be the SciCode resources server's venv, the only one with SciCode's scientific deps (`h5py`, `scipy`, `numpy`, `sympy`, `matplotlib`). In a split deployment — vLLM + the Ray head in one container, the resources server joining that cluster via `ray_head_node_address=auto` — the sub-step lands on a worker whose interpreter lacks those deps, so every generated program dies at `import h5py` and `mean/reward` is stuck at **0** even though the model's answers are fine. ## Fix Execute the sub-step **in the resources server's own process** via the thread-pool executor instead of shipping it to a Ray worker: ```python result = await loop.run_in_executor(None, run_substep, program, self.config.timeout_secs) ``` The resources server process already runs the SciCode venv, so `run_substep`'s `subprocess.run([sys.executable, ...])` now uses the interpreter that has the deps. Concurrency is unchanged — still bounded by the existing `num_processes` semaphore, and `subprocess.run` releases the GIL. The now-unused `run_substep_remote`/`import ray` are removed and the `verify()` tests retarget `run_substep`. ## Validation Ran the full SciCode benchmark end-to-end on a split (vLLM + separate resources) deployment. Before this change every sub-step failed at `import h5py` and reward was 0; after, sub-steps execute in the resources-server venv and `subtask_accuracy` lands in line with the NeMo-Skills SciCode reference. Unit tests updated and passing. ## Alternative considered Keeping the Ray task but pinning its interpreter (or a `runtime_env`) to the resources-server venv also works, but relies on that venv path being valid on every Ray worker (fragile across nodes/containers). Running in-process removes the ambient-interpreter assumption entirely; SciCode sub-steps are short, timeout-bounded subprocesses, so single-node execution is sufficient. Signed-off-by: laszkiewiczp <85294967+laszkiewiczp@users.noreply.github.com>
ECS Fargate `SandboxProvider` on top of the sandbox API (NVIDIA-NeMo#1377, now merged to `main`). Auto-mirrors public images to ECR on demand; SSH reverse-tunnel for exec / file-transfer / model egress. Rebased onto `main` now that the sandbox base (NVIDIA-NeMo#1377) has landed — this PR stands on its own and is ready for review. **Sandbox-bound agents line of work:** **ECS Fargate (this PR)** → adapter middleware base (NVIDIA-NeMo#1646) → sandbox-bound CLI agents + capture (NVIDIA-NeMo#1647). Interceptor follow-ups on the adapter base: NVIDIA-NeMo#1649 (caching), NVIDIA-NeMo#1650 (observability), NVIDIA-NeMo#1651 (rewrites). 3 commits, +5974: ECS provider + engine, on-demand ECR mirroring + conda activation fix, and 233 unit tests (engine.py to 99%). `uv.lock` reconciled with main's security upgrades (NVIDIA-NeMo#1657). --------- Signed-off-by: Michal Bien <mbien@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Currently, Gym reports only strict accuracy for ifbench. This PR intoduces a loose accuracy or ifbench (in addition to the strict accuracy) --------- Signed-off-by: Ewa Dobrowolska <edobrowolska@nvidia.com>
The _prefetch_judge_models() functions in benchmarks/longmt_pg19/prepare.py and benchmarks/longmt_wmt24pp/prepare.py were intended to pre-fetch models (comet, laser2, and ersatz) before agents were fanned out because those agents perform downloads which are not threadsafe. However, the venv that _prefetch_judge_models() is called from does not match that of the agents, causing the pre-fetch to fail and the subsequent downloads to collide. The agent venv is already fragile and I did not believe it was a good idea to duplicate those requirements across a second venv, potentially conflicting with existing requirements. Instead, I removed the _prefetch_judge_models() functions and added protection around each download within the agent code to prevent collision. I also added defaults for two environment variables (LASER_HOME, ERSATZ) in the agent environment only. A missing LASER_HOME in particular caused the code to crash. --------- Signed-off-by: Brian Thompson <3534106+thompsonb@users.noreply.github.com> Co-authored-by: Brian Thompson <3534106+thompsonb@users.noreply.github.com>
…IA-NeMo#1906) ## Summary Adds a built-in **`docker`** sandbox provider that runs each sandbox as a container on the **local Docker daemon** via the `docker` CLI — no OpenSandbox server or control plane required. It gives real container isolation out of the box on any machine with Docker, and plugs in behind the existing provider-neutral `SandboxProvider` protocol, so every agent built on `nemo_gym.sandbox` works with it unchanged. Part of the sandbox-providers epic (NVIDIA-NeMo#1685). ## Motivation The existing providers need either a remote control plane (`opensandbox`) or a non-Docker local runtime (`apptainer`, common on HPC). There was no zero-setup path for the most common developer/CI environment — a machine with Docker. This provider closes that gap; provider choice is a one-line config swap. ## Design Mirrors the `apptainer` provider's shape — a single subprocess chokepoint, a readiness probe, and neutral status/error mapping. **CLI-based, no new Python dependency.** ```mermaid flowchart LR Agent["Any agent"] --> API["nemo_gym.sandbox API"] API -- "create / exec / cp / close" --> P["DockerProvider"] P -- "docker run/exec/cp/rm" --> D[("local Docker daemon")] D --> C["sandbox container"] ``` - **One long-lived container per sandbox** — `docker run -d` + a keep-alive PID; `docker exec` for commands, `docker cp` for files, `docker rm -f` for teardown. State persists across `exec` calls. - **Config swap** — registers under `docker`; the shipped `docker.yaml` binds the same `sandbox` name every provider config uses, so switching is a single `+config_paths` change (no agent edits). - **`--entrypoint` override + `--init`** — the keep-alive is injected via `--entrypoint` so images with their own `ENTRYPOINT` (e.g. SWE-bench) don't swallow it; `--init` (tini) reaps short-lived `exec` processes. - **Auto-detected exec shell** — uses `bash` when the image has it (needed for conda `source`, e.g. SWE-bench), falls back to `sh`; resolved once per sandbox after readiness. Overridable via `exec_shell`. - **`ttl_s` self-destruct** — when set (no custom entrypoint), the keep-alive sleeps for `ttl_s` and `--rm` removes the container on exit — a safety net for leaked/expired sandboxes. - **Resource hard caps + isolation knobs** — `--cpus`, `--memory` (+ a matching `--memory-swap`), `--gpus <n>`; opt-in `network` (incl. `none`), `read_only`, `cap_drop`, `pids_limit`, `security_opt` (e.g. `no-new-privileges`); never `--privileged`. Containers are labeled for orphan reaping, and `--env` values are redacted from error messages. - **Per-sandbox `provider_options`** — `volumes` (→ `-v`) and `run_args`. ## Usage ```bash ng_run "+config_paths=[$AGENT, nemo_gym/sandbox/providers/docker/configs/docker.yaml, $MODEL]" ``` or directly: `Sandbox({"docker": {}}, spec)`. ## Docs Documented as a Fern provider page — `fern/versions/latest/pages/infrastructure/sandbox/docker.mdx` — a sibling of the Apptainer/OpenSandbox pages from NVIDIA-NeMo#1717 with the same shape (setup / config / `provider_options` / resource mapping / lifecycle / isolation), plus a Docker card in the sandbox index. The in-repo `README.md` stays as the developer reference, matching the other providers. ## Validation - **68 unit tests, 100% coverage** of the provider module — hermetic (mock the CLI boundary) with skip-guarded real-daemon tests. - Real-daemon smoke: create / exec / seeded files / upload / download / status / close; `ttl_s` self-destruct verified live. - **SWE-bench Verified (golden):** gold patch applied and graded in-sandbox → reward 1.0. - **SWE-bench Verified (agent):** a hosted OpenAI-compatible model drove `mini_swe_agent_2` end-to-end through the provider → resolved, reward 1.0. ## Acceptance criteria (NVIDIA-NeMo#1695) - [x] Provider implementing `SandboxProvider`, registered in `registry.py` - [x] YAML config + example, selectable via `sandbox_provider` - [x] Passing tests, skip-guarded when Docker/daemon is absent - [x] Docs — Fern provider page (sibling to the NVIDIA-NeMo#1717 sandbox pages) + repo README, covering setup + isolation/security - [x] Validated end-to-end on a real runtime (rollouts collected) ## Limitations - A command timeout returns a timeout result and leaves the reused sandbox running; the command's process is reaped with the container at `close()` (standard `docker rm -f` teardown). - Shares the host kernel — not an isolation boundary for untrusted code (use a VM/gVisor/Kata). Signed-off-by: Michal Bien <mbien@nvidia.com> Signed-off-by: Michał Bień <mbien@nvidia.com>
moves finance sec search environment into environments/ --------- Signed-off-by: Christian Munley <cmunley@nvidia.com>
## Summary - add absolute UTC generation and evaluation start timestamps to SWE rollout metrics - persist the OpenHands per-turn metrics block in `nemo_gym_metrics.json` - keep all new fields optional for compatibility with existing runs and consumers - preserve null evaluation timestamps when an agent produces no patch or evaluation never starts - pick the nv-OpenHands commit hash to reflect the changes from sdevare-nv/nv-OpenHands#19, it will provide more accurate timestamps. - upload `swe_trace_converter.py` file for profile visualization. ### Sample perfetto json visualization outcome <img width="2512" height="1183" alt="image" src="https://github.com/user-attachments/assets/b5c9a7ce-859d-48cf-82f1-cbcb767b13c5" /> ## Why The existing metrics file contained phase durations but no absolute phase anchors. Per-call OpenHands timing and token metrics were written only to `output_for_eval.jsonl`, forcing downstream tooling to join files and infer placement. With this change, `nemo_gym_metrics.json` is the complete per-instance timeline source. ## Validation - focused SWE-agent metrics tests: 6 passed, 105 deselected - OCI-HSG end-to-end Qwen3-30B-A3B SWE-rebench smoke: Slurm job `3652581`, completed `0:0` - generated metrics contained a parseable generation timestamp, 4 response records, 4 token records, and 4 action records - `per_turn_metrics` exactly matched `output_for_eval.jsonl.oh_time_metrics` - the intentional four-turn no-patch path correctly left `evaluation_start_timestamp` null Both commits are DCO signed. --------- Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
…NVIDIA-NeMo#1954) Two fixes needed to run tau3 through NEL: - Pin the tau2 `[knowledge]` extra deps (rank-bm25, openai) explicitly in requirements.txt — the extra is not applied to a `tau2[knowledge] @ git+URL` direct reference, so bm25/bm25_grep environments miss rank-bm25 on install. - Repoint the tau2-bench code pin (requirements.txt, source.py default, README) to `bxyu/nemo_gym_stable` and the data ref mention to `bxyu/nemo_gym_data` — the empty-user-message hard-fail handling (retry up to 30x, then fail the trajectory with `empty_user_message`) is now merged there (bxyu-nvidia/tau2-bench#20, NVIDIA-NeMo#17), superseding the personal dev branches. Verified end-to-end via NEL/EFB: 3x tau3-banking panda runs (8.04/9.69/7.01% vs 8.66% reference, within run-to-run noise) and qwen 8.87% (matches AA 8.7). Empty user responses occurred and recovered in-retry; zero hard-fails, zero ClientOSErrors. --------- Signed-off-by: Ewa Dobrowolska <edobrowolska@nvidia.com> Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Co-authored-by: Wojciech Prazuch <wprazuch@nvidia.com>
## Summary Adds [opencode](https://opencode.ai) as a second agent harness for `swe_agents`, alongside the existing OpenHands harness, plus several supporting benchmark and infrastructure additions: - **opencode harness** (`agent_framework: opencode`) — runs opencode's real `processor.ts` agentic loop via a custom `@opencode-ai/nemo-gym` `LanguageModelV3` provider that swaps the LLM transport while keeping opencode's tools, prompts, and parsing intact. Trajectory format, token-ID contiguity guarantees, and patch extraction are unchanged from OpenHands, so downstream training/eval code needs no changes to consume opencode rollouts. - **DeepSWE and DeNovoSWE dataset integrations** (⚠️ WIP, not yet baselined against a reference model) — new `DeepSWEDatasetProcessor` / `DeNovoSWEDatasetProcessor`, each with a golden-patch-validated harness (DeepSWE: 110/113 golden-resolved; DeNovoSWE: 3034/3668, 82.69%). - **swe-bench-ext fixes** for the variant-generation config path. - **Message replay for OpenHands** — a prior partial trajectory (system + user + `function_call`/`function_call_output` items) can be resumed instead of starting fresh, powering training-time replay and trajectory branching. - **Memory watchdog (OOM handling)** — the enroot/apptainer sandbox has no cgroup support, so per-container memory limits (`apptainer_memory_limit_mb`) are now enforced by a userspace RSS watchdog (`RunOpenHandsAgent._memory_watchdog`) instead of `ulimit -v`/`apptainer --memory` (both silently no-op in this sandbox). OOM kills feed into GRPO masking so accidental rewards aren't credited. - **Dataset-aware environment activation** for the opencode harness (SWE-Gym → conda `testbed`, R2E-Gym → `.venv`, default → conda `testbed`). - Results now write to a per-run dedicated folder instead of a shared directory. ## What's new ### opencode integration - New `OpenCodeHarnessProcessor`, paired with the existing `RunOpenHandsAgent.process_single_datapoint` driver (only the in-SIF command differs by `agent_framework`). - Workspace path resolution is dataset-aware (`_resolve_opencode_workspace_path`): defaults to `/testbed` (the real SWE-bench image convention) with per-dataset overrides for `nv-internal-1`/`deepswe` (`/app`), `denovoswe` (packed `workspace_path`), `swe-bench-ext` (`/workspace/repo`), and `SWE-rebench-V2` (`/{repo_name}`). - Subagent support behind `opencode_subagents_enabled`, with per-session trajectory capture and host-copy for offline training pickup. - Prompt overrides propagate the same way as OpenHands (`resolved_user_prompt_template` / `resolved_system_prompt_template`), rendered into a fixed SIF-mounted path. - Replay is intentionally **not** implemented for opencode (short-circuits to the OpenHands-only path). ### DeepSWE / DeNovoSWE (WIP) - DeepSWE: Harbor task format, in-SIF `tests/test.sh` verifier, reward read from `/logs/verifier/reward.txt`. Config: `configs/swebench_deepswe.yaml`. - DeNovoSWE: doc-to-repo tasks; wipes original source via `clean.sh`, re-injects the spec as `README.md`, applies the agent patch, lays canonical tests from `test_patch`, and runs per-file pytest with a `--collect-only` pre-flight. Config: `configs/swebench_denovoswe.yaml`. - Both are golden-patch validated but not yet baselined against a reference model — treat their reward signal as directional for now. ### Prompt, agent-class, and tool-name diversity - 15 prompt families under `prompts/` (breadth-first, codex, minimalist, root-cause, etc.), each pairing a `system_prompt.j2`/`user_prompt.j2` with an `agent_cls` (`CodeActAgent`, `OpenCodeAgent`, `CodexAgent`, `Terminus2Agent`). - Selection is deterministic per `instance_id` by default (reproducible eval) or uniformly random per run when `agent_prompt_override_random=true` (RL training diversity). - Independent `diversify_tool_names` / `camel_case_tool_names` knobs to break the model's reliance on memorized tool names. ### Reliability / infra - Memory watchdog replaces the (nonfunctional in this sandbox) cgroup/ulimit-based memory limiting; OOM kills on either the agent or eval container are tracked and mask the GRPO sample. - Message replay for OpenHands, with the input/output split in the response found by matching tool-call ids against the replayed prefix. - Ground-truth redaction: only an allowlisted set of instance fields (`AGENT_VISIBLE_INSTANCE_FIELDS`) is exposed inside the agent container; eval-only fields (gold/test patches, graded test names) stay host-side under `eval_private/`. ### GRPO masking (`mask_sample`) `SWEBenchWrapperInstanceConfig.mask_sample` tells downstream RL to drop the gradient for a rollout whenever the reward can't be trusted, rather than filtering it out entirely — the trajectory is still returned, just flagged. It's set to `True` when any of: 1. The patch resolved the tests **but** the agent didn't actually submit cleanly — it hit `max_iteration` or blew the `context_window` (bucketed from the agent's error string by `_classify_agent_error`) — the reward is accidental, not earned. 2. The eval container hit `swebench_tests_timeout` — the reward is unreliable (tests may not have finished running). 3. The agent itself hit `swebench_agent_timeout` (wall-clock) — masked regardless of whether it resolved. 4. The memory watchdog OOM-killed the **agent** container. 5. The memory watchdog OOM-killed the **eval** container. --------- Signed-off-by: Sugam Devare <sdevare@nvidia.com> Signed-off-by: Junkeun Yi <jkyi@nvidia.com> Co-authored-by: Junkeun Yi <jkyi@nvidia.com>
Signed-off-by: Sugam Devare <sdevare@nvidia.com>
## Summary
The MCQA verifier's `strict_single_letter_boxed` grading mode only
accepted a single uppercase letter directly inside `\boxed{...}`. Real
model rollouts frequently emit the answer wrapped in `\text{}` (e.g.
`\boxed{\text{E}}`) or with the option text appended after the letter
(e.g. `\boxed{E: A polygenic risk score...}`), and these were
incorrectly scored as `no_answer`. This PR rewrites the boxed-letter
extraction to tolerate those formats while preserving strict semantics,
recovering rewards that were previously lost on valid answers.
## Key Changes
- Replaced the single regex (`STRICT_BOXED_PATTERN`) with a
brace-counting helper `_extract_boxed_inner()` that returns the full
contents of the last `\boxed{...}`, correctly handling nested braces
like `\boxed{\text{E}}` instead of stopping at the first `}`. Selecting
the last box means CoT intermediate boxes (e.g. `\boxed{C} ...
\boxed{E}`) no longer shadow the final answer, matching the convention
in `math_with_code`/`abstention`.
- Strict extraction now strips `\text{...}` wrappers, then matches
either a bare padded letter (`BOXED_LETTER_ONLY_PATTERN`) or a leading
letter followed by a label delimiter (`:`, `)`, `.`, `-`) with trailing
option text ignored (`BOXED_LETTER_LABEL_PATTERN`).
- Strict mode deliberately does not match bare option text without a
leading letter - that remains `lenient_boxed`'s responsibility -
preserving the boundary between the two modes.
- `_match_option_text` (lenient mode) reuses the same
`_extract_boxed_inner()` helper, so lenient inherits both the
nested-brace and last-box handling.
- README grading-mode docs updated to describe the new tolerated
formats, last-box selection, and the explicit non-match case.
## Validation
Validated the extractor directly (in isolation from model sampling)
across the boxed-answer formats seen in real rollouts from
gpt-4.1-2025-04-14:
- **Recovered formats** - `\boxed{\text{E}}`, `\boxed{E: <option
text>}`, and `\boxed{\text{C: <option text>}}` now resolve to their
letter instead of scoring `no_answer` (the formats models emit when they
ignore the "letter and nothing else" prompt).
- **Last box wins** - with multiple `\boxed{...}` (e.g. a CoT
intermediate before the final answer), the extractor reads the last one;
verified for both a letter intermediate (`\boxed{C} ... \boxed{E}` ->
`E`) and a non-letter one (`\boxed{\text{idea}} ... \boxed{E}` -> `E`).
- **Strict stays strict** - bare option text with no leading letter
(`\boxed{A polygenic...}`) is deliberately left unmatched, preserving
the `strict` / `lenient_boxed` boundary.
- **Fails safe** - an unbalanced `\boxed{` returns `no_answer`, not a
crash.
- **Lenient inheritance** - `lenient_boxed` picks up the same `\text{}`
and nested-brace handling through the shared helper.
| Boxed output | `main` | this branch |
|---|---|---|
| `\boxed{\text{E}}` | `no_answer` | `E` ✓ |
| `\boxed{\text{B: Polymerase Chain Reaction (PCR)}}` | `no_answer` |
`B` ✓ |
| `\boxed{E: \text{A polygenic risk score...}}` | `no_answer` | `E` ✓ |
| `\boxed{\text{I: NGS can detect...}}` | `no_answer` | `I` ✓ |
---------
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>
…VIDIA-NeMo#1234) ## Summary Two related changes to harmonize judge-side knobs between rubric and comparison modes: 1. **Rubric default max_tokens 8192 → 65535** (`scoring.py:score_with_rubric`, `score_with_rubric_visual`). Matches the default already in place for comparison mode (`comparison.send_judge_request`) and structured rubric (`score_with_rubric_structured`). Prevents `finish_reason: length` truncations on rubrics with many criteria when callers don't override. 2. **`create_overrides` plumbing for comparison-mode judge calls** — adds the same override mechanism to `comparison.send_judge_request` and threads it through `run_trials` and `_verify_comparison`. Mirrors the rubric path where `judge_responses_create_params_overrides` already gets merged into `chat.completions.create`. Also switches `_verify_comparison` from `overrides.get` → `overrides.pop` so `model`/`api_key` are consumed and the rest pass through (same pattern `_verify_rubric` uses post NVIDIA-NeMo#1225). After this lands, callers can override max_tokens / temperature / top_p / etc. for the comparison-mode judge from Hydra config exactly the same way they already can for rubric mode. ## Replaces NVIDIA-NeMo#1215 Supersedes NVIDIA-NeMo#1215 (same change, rebased onto current main). The original branch was DIRTY against main due to overlap with NVIDIA-NeMo#1225 (raw_responses kwarg on `run_trials`), NVIDIA-NeMo#1228 (preconvert + `clean_up_list`), and NVIDIA-NeMo#1229 (`JUDGE_REQUEST_TIMEOUT_SECONDS`). All three are kept; this PR adds `create_overrides` alongside. ## Test plan - [x] Existing `test_app.py::test_verify_rubric_passes_create_overrides_through` continues to verify the rubric path. - [ ] Ad-hoc smoke: run a 5-task comparison-mode rollout with `++gdpval_judge_model.responses_api_models.openai_model.max_tokens=8192` to verify the override actually reaches the comparison judge call. Signed-off-by: Serge Panev <spanev@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on NVIDIA-NeMo#1377.
Adds a Daytona-backed sandbox provider to the sandbox API provider registry, including SDK dependency wiring and provider lifecycle/exec/file helpers.
Smoke notes:
Total disk limit exceeded. Maximum allowed: 30GiB), so the observed 1/16 reward is not a meaningful model-quality pass rate.sandbox-api: daytona-sdk.