Improve Github SSH Key setup docs by bxyu-nvidia · Pull Request #12 · NVIDIA-NeMo/Gym

bxyu-nvidia · 2025-09-05T23:12:53Z

No description provided.

Signed-off-by: Brian Yu <bxyu@nvidia.com>

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: soares-f <soarescmsa@gmail.com>

Signed-off-by: Brian Yu <bxyu@nvidia.com>

Supersedes #1510; covers epic #1205 friction #8 + #12 and issues #1488, #1489, #1490 in one place: - #1488: missing config_paths entry -> ConfigPathNotFoundError (names entry + searched locations). - #1490: malformed (non-list) config_paths -> MalformedConfigPathsError with the expected Hydra list syntax. - #1489: zero configured servers -> NoServerInstancesError, raised in RunHelper.start() before Ray (covers ng_run AND e2e_rollout_collection). All three subclass a new ConfigError base. A CLI decorator (exit_cleanly_on_config_error) on run()/e2e_rollout_collection() turns any ConfigError into a clean, rich-escaped message + exit 1 with NO traceback (the explicit ask in #1488/#1489), while keeping them ordinary exceptions so ng_validate (#1599) can still catch and format them. Zero-server check uses validated server instances, not a raw key count. Tests: deterministic tmp_path-based path-error tests (both-locations, dedup, absolute), malformed-config_paths, zero-server, and the decorator (ConfigError -> clean exit; non-ConfigError propagates). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Ports ng_validate into the unified gym CLI (#1434): a validate() command in cli/env.py registered as 'gym env validate' (+ ng_validate/nemo_gym_validate deprecated shims). Runs the full parse with no Ray, exits 0/1 with a clean, rich-escaped message. Targets martas/1434. Epic #1205 friction #12 (no config validation tooling). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Ports #1609 onto the unified gym CLI (#1434). Covers epic #1205 friction #8 + #12 and issues #1488/#1489/#1490: - #1488: missing config_paths entry -> ConfigPathNotFoundError - #1490: malformed (non-list) config_paths -> MalformedConfigPathsError - #1489: zero configured servers -> NoServerInstancesError, in RunHelper.start() before Ray (covers env run + eval run / e2e) All subclass ConfigError; a cli/env.py decorator (exit_cleanly_on_config_error, also applied to e2e_rollout_collection in cli/eval.py) turns them into a clean, rich-escaped message + exit 1, no traceback. Targets martas/1434. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Ports ng_validate into the unified gym CLI (#1434): a validate() command in cli/env.py registered as 'gym env validate' (+ ng_validate/nemo_gym_validate deprecated shims). Runs the full parse with no Ray, exits 0/1 with a clean, rich-escaped message. Targets martas/1434. Epic #1205 friction #12 (no config validation tooling). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

Ports #1609 onto the shared CLI base (martas/1434 + uv pin + concurrency). Covers epic #1205 friction #8 + #12 and issues #1488/#1489/#1490: ConfigPathNotFoundError, MalformedConfigPathsError, NoServerInstancesError (all ConfigError); fail-fast guard in RunHelper.start(); exit_cleanly_on_config_error decorator on run()/e2e_rollout_collection() -> clean message, no traceback. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…hs (#1205 #8/#12; #1488/#1489/#1490) (#1609) ## What One coherent fix for the three `config_paths` failure modes, all failing fast with a **clean message and no traceback**: | Issue | Bad input | Before | After | |-------|-----------|--------|-------| | [#1488](#1488) | typo'd / missing path | 7-frame `FileNotFoundError` | `Error: config_paths entry '...' was not found. Looked in: ...` | | [#1490](#1490) | scalar instead of list | raw Pydantic `ValidationError` (+pydantic.dev URL) | `Error: 'config_paths' must be a list of paths ...` | | [#1489](#1489) | empty / omitted (zero servers) | Ray starts, hangs until SIGTERM | `Error: No server instances are configured ...` before Ray | ## How - New `ConfigError` base in `config_types.py`; `ConfigPathNotFoundError`, `MalformedConfigPathsError`, `NoServerInstancesError` (and the existing `ServerRefNotFoundError`) inherit it. They stay ordinary exceptions, so `ng_validate` (#1599, `except Exception`) can still catch and format them. - `load_extra_config_paths` wraps `OmegaConf.load` (#1488); `parse()` wraps `ta.validate_python` (#1490). - `raise_on_no_server_instances` (validated server-instance check, not a raw key count) runs in **`RunHelper.start()`** before `initialize_ray()` — covers both run and `e2e_rollout_collection` (#1489). - A CLI decorator `exit_cleanly_on_config_error` converts any `ConfigError` to a rich-escaped message + `exit 1`, **no traceback**. Unexpected errors still propagate unchanged. ## Supersedes #1510 Consolidates [#1510](#1510): typed exceptions `ng_validate` can catch/format (a bare `SystemExit` would escape it), a validated zero-server check, the guard in `start()` so `e2e` is covered, and deterministic tests. ## Tests `test_global_config.py`: path-not-found (both-locations / dedup / absolute), malformed config_paths, zero-server (raise + pass). `test_cli.py`: the decorator (ConfigError → clean exit; non-ConfigError propagates unchanged; success passes through). 67/67 `test_global_config`+`test_cli` pass; ruff + pre-commit clean. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…ction #12 / FEP-1016) Add `gym env validate` (+ ng_validate/nemo_gym_validate deprecated shims): run the full config parse with no Ray and no server subprocesses, exit 0 (valid) / 1 (invalid) with a clean, traceback-free message — config errors otherwise only surface ~30-60s later after Ray bootstrap. It reuses get_global_config_dict so the checks stay in sync: config_paths resolution (#1488/#1490), server cross-references (#1561), mandatory ??? values (#1575), and schema. A dummy policy_model (NO_MODEL) is injected so model interpolations resolve without real creds — the model is supplied by --model* at run time. Registered in the env group with the same config-selection flags as env start; wrapped in exit_cleanly_on_config_error. Tests: env validate routing + validate() valid/invalid behavior. Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…tion #12) (#1599) ## What Adds **`gym env validate`** (+ `ng_validate` / `nemo_gym_validate` deprecated shims) — runs the full config parse with **no Ray and no server subprocesses**, then exits **0 (valid) / 1 (invalid)** with a clean, rich-escaped message (**no traceback**). Returns in well under a second instead of after a ~30–60s Ray bootstrap. ```bash gym env validate --config resources_servers/<env>/configs/<env>.yaml --config responses_api_models/<model>/configs/<model>.yaml gym env validate --benchmark gsm8k --model-type openai_model ``` ## How `validate()` lives in `cli/env.py` and is registered as `env validate` in the `gym` router (`cli/main.py` COMMANDS) with the same config-selection flags as `env start` (`--config`, `--benchmark`, `--environment`, `--resources-server`, `--model-type`, `--search-dir`, `--model*`). It reuses the same `get_global_config_dict()` parse path the other commands use, so the validation checks stay in sync: - **config_paths** resolution — missing/typo'd ([#1488](#1488)) and malformed ([#1490](#1490)) - **server cross-references** — unknown `name:` refs ([#1561](#1561)) - **mandatory `???`** values ([#1575](#1575)) - **schema** (`BaseNeMoGymCLIConfig`) Wrapped in `exit_cleanly_on_config_error` (from #1609) so any `ConfigError` becomes a clean message + `exit 1`. A dummy `policy_model` is injected (the `NO_MODEL` parser config, as in `gym list` / `env compose`) so model interpolations like `${policy_base_url}` resolve without real creds — validation is about config **well-formedness**; the real model is supplied by the `--model*` flags at run time. ## Targets `main` Originally drafted on the unified-CLI epic branch; rebuilt directly on `main` now that [#1630](#1630) (and #1637/#1609/#1635/#1671) have merged. The old branch contents (a snapshot of the CLI refactor + unrelated CI commits) were superseded and replaced. ## Scope note The zero-server check ([#1489](#1489), "nothing configured to run") is intentionally **not** part of `validate`: `NO_MODEL` injects a dummy model server (which would defeat the check), and "is anything configured to run" is a *start*-time concern already enforced by `gym env start` before Ray init. `validate` focuses on config well-formedness. ## Why Epic [#1205](#1205) friction #12 (no config validation tooling) — the M1 "fast failure triage" deliverable. Config errors otherwise only surface after Ray starts (~30–60s). ## Tests - `test_cli_main.py`: `gym env validate --config X` routes to `nemo_gym.cli.env:validate` with `+config_paths=[X]` (added to the parametrized config-command matrix). - `test_cli.py`: `validate()` prints OK on a valid config; a raised `ConfigError` becomes `exit 1` (no traceback). - All `test_cli` + `test_cli_main` + `test_cli_legacy` pass (the only failures are the pre-existing Python-3.12 `TestDidYouMean` argparse issue on `main`); ruff + pre-commit clean. Smoke-tested end-to-end: `✓ Config is valid.` on a real benchmark, clean error + `exit 1` on a bad path, and the `ng_validate` deprecation shim. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>

…hs (#1205 #8/#12; #1488/#1489/#1490) (#1609) ## What One coherent fix for the three `config_paths` failure modes, all failing fast with a **clean message and no traceback**: | Issue | Bad input | Before | After | |-------|-----------|--------|-------| | [#1488](#1488) | typo'd / missing path | 7-frame `FileNotFoundError` | `Error: config_paths entry '...' was not found. Looked in: ...` | | [#1490](#1490) | scalar instead of list | raw Pydantic `ValidationError` (+pydantic.dev URL) | `Error: 'config_paths' must be a list of paths ...` | | [#1489](#1489) | empty / omitted (zero servers) | Ray starts, hangs until SIGTERM | `Error: No server instances are configured ...` before Ray | ## How - New `ConfigError` base in `config_types.py`; `ConfigPathNotFoundError`, `MalformedConfigPathsError`, `NoServerInstancesError` (and the existing `ServerRefNotFoundError`) inherit it. They stay ordinary exceptions, so `ng_validate` (#1599, `except Exception`) can still catch and format them. - `load_extra_config_paths` wraps `OmegaConf.load` (#1488); `parse()` wraps `ta.validate_python` (#1490). - `raise_on_no_server_instances` (validated server-instance check, not a raw key count) runs in **`RunHelper.start()`** before `initialize_ray()` — covers both run and `e2e_rollout_collection` (#1489). - A CLI decorator `exit_cleanly_on_config_error` converts any `ConfigError` to a rich-escaped message + `exit 1`, **no traceback**. Unexpected errors still propagate unchanged. ## Supersedes #1510 Consolidates [#1510](#1510): typed exceptions `ng_validate` can catch/format (a bare `SystemExit` would escape it), a validated zero-server check, the guard in `start()` so `e2e` is covered, and deterministic tests. ## Tests `test_global_config.py`: path-not-found (both-locations / dedup / absolute), malformed config_paths, zero-server (raise + pass). `test_cli.py`: the decorator (ConfigError → clean exit; non-ConfigError propagates unchanged; success passes through). 67/67 `test_global_config`+`test_cli` pass; ruff + pre-commit clean. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>

## What does this PR do? Adds two capabilities to the SWE agent harness (`responses_api_agents/swe_agents/`), plus fixes to the DeepSWE verifier path found while baselining that dataset. **1. opencode patch mode.** New `opencode_patch_mode` config field (`worktree` | `committed`, default `worktree`), exported to the harness as `PATCH_MODE` and forwarded to the bench CLI as `--patch-mode`. `worktree` is the existing capture — `git diff` of the working tree with untracked files marked intent-to-add — and is unchanged. `committed` instead diffs the pre-run HEAD against the most advanced commit the agent left behind, searched across HEAD and every local branch. That mode is required by task families whose problem statement tells the agent to commit its solution: DeepSWE's statements end with *"work on this in a new branch from main and commit everything when you are done"*, so those rollouts finish with a clean tree and `git diff` records every patch as 0 bytes regardless of whether the model solved the task. Both modes emit a plain `base -> final tree` unified diff, so the eval side is unchanged. **2. opencode trajectory replay.** New `opencode_replay.py` module plus wiring, so a rollout can resume a partially-completed trajectory on a fresh container instead of restarting the task. When a request's `input` carries a prior trajectory (`function_call` / `function_call_output` items beyond the seed messages), it is converted to chat-completion format and surfaced as `problem_info["replay_messages"]`; the processors materialize it (plus the recorded system prompt and, for opencode, a subagent manifest) and forward the paths to `run_infer.sh` as positional args (opencode `#13`/`#14`, openhands `#18`). Recorded subagent sessions are linked to the exact parent task call that spawned them rather than by metadata order, so parallel siblings and nested agents can't consume each other's turns, and live continuations are merged back onto the recorded root. Per-session trajectory records now also carry replay linkage and global ordering (`recorded_session_id`, `spawn_call_id`, `spawn_index`, `global_turn`, …), and `SWEBenchVerifyResponse.subagent_trajectories` is populated so returned create-params are directly replay-ready. **3. DeepSWE verifier fixes.** - Carry optional `tests/grader.py` / `tests/config.json` from Harbor bundles (DeepSWE v1.1 delegates preparation and scoring to them) and mount them into the eval container. Optional, so older bundles still convert and grade. - Read the reward from `/logs/verifier/reward.json` (new tasks) as well as `reward.txt` (older tasks and crash sentinels). - Fix `mkdir -p /logs/vserifier` → `/logs/verifier`; the typo meant the directory the verifier writes its reward into was never created, so the reward read back empty and the task graded 0. - Skip injecting the online Maven mirror for verifiers that deliberately run offline (`mvn -o` / `--offline`) against their image's build-time JVM cache — Maven records which repository supplied each cached artifact, so a differently-named mirror makes cached plugins look unavailable. **4. Reverts #2120** (SWE agent rollout observations), removing `observability.py`, `tests/test_observability.py` and their call sites.  Tests: `responses_api_agents/swe_agents/tests/test_app.py` gains 26 tests covering the replay path — gym-side message conversion (openhands and opencode), replay file materialization and system-prompt pinning, positional-arg ordering, subagent manifest mounting with `ENABLE_SUBAGENTS=1`, parent-task-call linkage for parallel and nested children, completed-invocation truncation, legacy payload parsing, live-continuation merging, and preservation of the new trajectory record fields. Note: end-to-end opencode replay also needs `--replay-messages-file` / `--replay-subagents-file` in the opencode fork. The commit currently pinned by `configs/swebench_opencode.yaml` (`nv-opencode@sdd/dev`) does not have them — its `run_infer.sh` stops at positional `#12` — so until that lands and the pin moves, an opencode replay request is a no-op and the agent starts the task from scratch. openhands replay is unaffected. Bumping the pin also requires clearing `cache/swe_agents/swe_opencode_setup`, since `OpenCodeHarnessProcessor.setup()` returns early once the tree exists and never re-checks-out the configured commit. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Sugam Devare <sdevare@nvidia.com>

impl

4842d84

Signed-off-by: Brian Yu <bxyu@nvidia.com>

copy-pr-bot Bot temporarily deployed to test September 5, 2025 23:12 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci September 5, 2025 23:13 Inactive

bxyu-nvidia merged commit b2e5300 into main Sep 5, 2025

bxyu-nvidia deleted the bxyu/improve-github-ssh-key-setup branch September 5, 2025 23:23

soares-f pushed a commit that referenced this pull request Sep 8, 2025

Improve Github SSH Key setup docs (#12)

a1cff4b

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: soares-f <soarescmsa@gmail.com>

soares-f pushed a commit that referenced this pull request Sep 15, 2025

Improve Github SSH Key setup docs (#12)

656911f

Signed-off-by: Brian Yu <bxyu@nvidia.com> Signed-off-by: soares-f <soarescmsa@gmail.com>

abubakaria56 pushed a commit to abubakaria56/Gym that referenced this pull request Mar 2, 2026

Improve Github SSH Key setup docs (NVIDIA-NeMo#12)

8c753d1

Signed-off-by: Brian Yu <bxyu@nvidia.com>

abubakaria56 pushed a commit to abubakaria56/Gym that referenced this pull request Mar 2, 2026

Improve Github SSH Key setup docs (NVIDIA-NeMo#12)

922768e

Signed-off-by: Brian Yu <bxyu@nvidia.com>

This was referenced Jun 15, 2026

feat(cli): add 'gym env validate' pre-flight config check (#1205 friction #12) #1599

Merged

feat(config): unified clean errors for bad/malformed/empty config_paths (#1205 #8/#12; #1488/#1489/#1490) #1609

Merged

ritaneves mentioned this pull request Jun 25, 2026

[Epic] Continuation of configuration works #1726

Closed

ritaneves mentioned this pull request Jun 25, 2026

epic: reduce configuration friction for environment authors and users #1205

Closed

5 tasks

willtong-aq mentioned this pull request Sep 23, 2026

Add critpt_custom_grader: a sandboxed, locally executed grader for CritPt-style problems #3430

Closed

AdamRajfer mentioned this pull request Oct 9, 2026

feat(gdpval): raise per-call output and episode limits in sandbox recipes AdamRajfer/Gym#2

Closed

Navigation Menu

Repository navigation

Improve Github SSH Key setup docs - #12

Improve Github SSH Key setup docs#12
bxyu-nvidia merged 1 commit into
mainfrom
bxyu/improve-github-ssh-key-setup

bxyu-nvidia commented Sep 5, 2025

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bxyu-nvidia commented Sep 5, 2025

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant