Skip to content

cp: docs(cli): fix cli-commands.mdx issues from NVBug 6854088 (4093) into r0.7.0 #4219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
marta-sd merged 1 commit into from
Oct 8, 2026
Merged

cp: docs(cli): fix cli-commands.mdx issues from NVBug 6854088 (4093) into r0.7.0 #4219

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 90 additions & 13 deletions fern/versions/latest/pages/reference/cli-commands.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,15 @@ gym env validate # validate a manifest-backed workload or legacy co
gym env packages # list packages in a server's virtual environment
gym env test # run resources-server tests
gym env publish # finalize readiness and confirm registry discovery
gym env prefetch # pre-warm per-server venvs without starting servers
gym env start # start the servers
gym env status # show running servers
gym env schema # print a resources server's task-data schema as JSON Schema

# Evaluation
gym eval prepare # prepare benchmark data and dump it to disk
gym eval run # collate data, start servers, and collect rollouts
gym eval submit # submit a benchmark job to Slurm
gym eval aggregate # merge sharded rollout results
gym eval export # export supported Gym trajectories as ATIF
gym eval health-check # verify rollout artifact quality for an existing run
Expand Down Expand Up @@ -91,6 +94,9 @@ Commands that need a model (`gym env start`, `gym eval run`) configure it throug
| `--model-url` | Base URL of an existing model server endpoint. Maps to `policy_base_url`. |
| `--model-api-key` | API key for the model server. Maps to `policy_api_key`. |

<Warning>
`--model-api-key` passed on the command line appears in shell history. Prefer setting `policy_api_key` in your `env.yaml` instead.
</Warning>

### Swapping the agent

Expand Down Expand Up @@ -324,6 +330,10 @@ gym search models vllm --json

Commands for preparing, previewing, and managing datasets. By default dataset transfer commands use HuggingFace; pass `--storage gitlab` to target the GitLab Registry.

<Note>
HuggingFace commands require an `HF_TOKEN` environment variable (or `huggingface-cli login`). GitLab commands authenticate via `mlflow_tracking_token` in your YAML config (or the `MLFLOW_TRACKING_TOKEN` environment variable). See [Downloading from HuggingFace](/data/download-huggingface) for HF authentication details.
</Note>

### `gym dataset upload`

Upload a prepared local JSONL dataset to HuggingFace (default) or GitLab.
Expand Down Expand Up @@ -421,6 +431,7 @@ Validate and collate a dataset, generating metrics and statistics.
| --- | --- |
| `--config PATH` | Config file to load. Repeatable. |
| `--resources-server NAME` | Load the named resources server config. |
| `--model-type NAME` | Load the named model-type config. |
| `--search-dir DIR` | Extra root directory to search for named components. Repeatable. |
| `--mode {train_preparation,example_validation}` | Use `train_preparation` to prepare train/validation datasets, or `example_validation` to validate example data. |
| `--output-dir` | Output directory for the prepared data. |
Expand Down Expand Up @@ -528,7 +539,7 @@ By default, values a run needs but the config leaves undefined (such as `policy_
| `--model-type NAME` | Also load a named model config (otherwise a dummy `policy_model` is used). |
| `--search-dir DIR` | Extra root directory to search for named components. Repeatable. |
| `--agent-type NAME[/FLAVOR]` | Compose the environment with the named agent harness. Refer to [Swapping the agent](#swapping-the-agent). |
| `--allow-unsupported-pairing` | Run even if the resources server does not declare support for the selected agent. |
| `--allow-unsupported-pairing` | Run even if the environment's resources server does not declare support for the selected agent or model type. |
| `--model` / `--model-url` / `--model-api-key` | Override model name, base URL, and API key. |

```bash
Expand Down Expand Up @@ -598,6 +609,7 @@ Run the local publication check for a manifest-backed workload. The command runs
| `NAME` | Manifest-backed catalog entry to publish. |
| `--kind environment\|benchmark` | Resolve an ambiguous catalog name. |
| `--json` | Output the publication result as JSON. |
| `--search-dir DIR` | Extra root directory to search for named components. Repeatable. |

```bash
gym env publish my_eval
Expand All @@ -606,17 +618,18 @@ gym env publish my_benchmark --kind benchmark --json

### `gym env start`

Start the NeMo Gym servers (agents, models, resources) defined by the provided configs. Reads configuration from YAML files and runs each configured server in its own environment.
Start the NeMo Gym servers (agents, models, resources, and environment servers) defined by the provided configs. Reads configuration from YAML files and runs each configured server in its own environment. Rollout collection dispatches `/run` through environment servers.

| Option | Description |
| --- | --- |
| `--config PATH` | Config file to load. Repeatable. |
| `--benchmark NAME` | Load the named benchmark config (start its servers). |
| `--environment NAME` | Load the named environment config. Repeatable. |
| `--resources-server NAME` | Load the named resources server config. |
| `--model-type NAME` | Load the named model server type config. |
| `--model-type NAME` | Load the named model-type config. |
| `--search-dir DIR` | Extra root directory to search for named components. Repeatable. |
| `--agent-type NAME[/FLAVOR]` | Compose the environment with the named agent harness. Refer to [Swapping the agent](#swapping-the-agent). |
| `--allow-unsupported-pairing` | Run even if the resources server does not declare support for the selected agent. |
| `--allow-unsupported-pairing` | Run even if the environment's resources server does not declare support for the selected agent or model type. |
| `--model`, `-m` | Served model identifier. See [Selecting the model server](#selecting-the-model-server). |
| `--model-url` | Model server base URL. |
| `--model-api-key` | Model server API key. |
Expand Down Expand Up @@ -660,6 +673,40 @@ NeMo Gym Server Status:
3 servers found (3 healthy, 0 unhealthy)
```

### `gym env prefetch`

Pre-warm per-server venvs without starting any servers. Accepts the same config flags as `gym env start`. For each server in the config, creates its isolated venv and installs dependencies serially. Intended for use in Dockerfile builds so venvs are ready at runtime without network access.

| Option | Description |
| --- | --- |
| `--config PATH` | Config file to load. Repeatable. |
| `--benchmark NAME` | Load the named benchmark config. Repeatable. |
| `--environment NAME` | Load the named environment config. Repeatable. |
| `--resources-server NAME` | Load the named resources server config. Repeatable. |
| `--model-type NAME` | Load the named model-type config. |
| `--agent-type NAME[/FLAVOR]` | Compose with the named agent harness. |
| `--allow-unsupported-pairing` | Run even if the environment's resources server does not declare support for the selected agent or model type. |
| `--search-dir DIR` | Extra root directory to search for named components. Repeatable. |

```bash
config_paths="responses_api_models/local_vllm_model/configs/local_vllm_model.yaml,\
resources_servers/math/configs/math.yaml"
gym env prefetch "+config_paths=[${config_paths}]"
```

### `gym env schema`

Print a resources server's `TaskData` schema as JSON Schema. Exits non-zero if the server does not export a `task_data.py` schema.

| Option | Description |
| --- | --- |
| `--resources-server NAME` | Resources server to inspect. |
| `--search-dir DIR` | Extra root directory to search for named components. Repeatable. |

```bash
gym env schema --resources-server example_multi_step
```

---

## Evaluation
Expand Down Expand Up @@ -689,14 +736,15 @@ Collate data, start the servers, and collect rollouts. This is the main evaluati
| --- | --- |
| `--config PATH` | Config file to load. Repeatable. |
| `--benchmark NAME` | Load the named benchmark config. |
| `--environment NAME` | Load the named environment config. Repeatable. |
| `--resources-server NAME` | Load the named resources server config. |
| `--model-type NAME` | Load the named model server type config. |
| `--model-type NAME` | Load the named model-type config. |
| `--search-dir DIR` | Extra root directory to search for named components. Repeatable. |
| `--no-serve` | Collect against already-running servers instead of starting them. |
| `--resume` | Resume from cached rollouts instead of recollecting. Maps to legacy `+resume_from_cache=true`. Refer to [Resume interrupted runs](#resume-interrupted-runs). |
| `--agent-type NAME[/FLAVOR]` | Compose the environment with the named agent harness. Rejected with `--no-serve`. Refer to [Swapping the agent](#swapping-the-agent). |
| `--agent`, `-a` | Agent instance to collect rollouts with. |
| `--allow-unsupported-pairing` | Run even if the resources server does not declare support for the selected agent. |
| `--allow-unsupported-pairing` | Run even if the environment's resources server does not declare support for the selected agent or model type. |
| `--input`, `-i` | Input tasks JSONL file. |
| `--output`, `-o` | Output rollouts JSONL file. |
| `--limit` | Maximum number of tasks to run. |
Expand Down Expand Up @@ -724,21 +772,28 @@ gym eval run --benchmark aime24 \
--split validation \
--concurrency 10

# Against an already-running server, with a remote vLLM endpoint
gym eval run --no-serve \
# Two-terminal sequence for --no-serve: model settings go to env start, not eval run.
# Model URL and credentials configure servers at start time; passing them during
# collection has no effect on already-running servers.

# Terminal 1 — start servers with the model endpoint configured:
gym env start \
--resources-server math_with_judge \
--model-type openai_model \
--model openai/gpt-oss-120b \
--model-url http://127.0.0.1:10240/v1 \
--model-api-key dummy_key

# Terminal 2 — collect rollouts against the running servers:
gym eval run --no-serve \
--resources-server math_with_judge \
--output results/test_001.jsonl \
--split validation \
--model openai/gpt-oss-120b \
--model-url http://0.0.0.0:10240/v1 \
--model-api-key dummy_key \
--temperature 1.0 \
--top-p 1.0

# Per-agent repeats: one input file pins different agents per row via agent_ref.name
gym eval run --no-serve \
--model-type openai_model \
--input mixed_agents.jsonl \
--output results/mixed_rollouts.jsonl \
--num-repeats '{agent_alpha: 4, agent_beta: 1, _default: 1}'
Expand Down Expand Up @@ -787,6 +842,23 @@ How it works:
If you change the config, schema, or data between runs, the materialized inputs become stale and resume will diff against the old expansion. Delete the `*_materialized_inputs.jsonl` file (and the output file) to start fresh.
</Warning>

### `gym eval submit`

Submit a benchmark evaluation job to Slurm via a submit config YAML file. Renders job scripts and calls `sbatch` for each benchmark defined in the config. Use `--dry-run` to preview the generated scripts without submitting.

| Option | Description |
| --- | --- |
| `--config PATH`, `-c PATH` | Submit config YAML file. Required. |
| `--dry-run` | Print generated job scripts without submitting. |
| `--resolve-only` | Resolve and validate the config, print it, and stop before rendering job scripts or submitting anything. |
| `--json` | Emit the submission record (or, with `--resolve-only`, the resolved config) as JSON. |

```bash
gym eval submit --config submit.yaml
gym eval submit --config submit.yaml --dry-run
gym eval submit --config submit.yaml --resolve-only --json
```

### `gym eval aggregate`

Merge sharded rollout results into a single rollouts file with aggregate metrics. Reads every JSONL file matching `--input-glob`, recomputes aggregate metrics over the global union of records, and writes a `<output stem>_aggregate_metrics.json` next to the merged rollouts. Use this to combine shards produced by `gym eval run --no-serve --disable-aggregation`.
Expand Down Expand Up @@ -817,7 +889,7 @@ omitting data. All invalid JSONL rows are reported together, and no destination
model attempts not selected by a turn, such as failed retries, remain in `extra.nemo_gym.surplus_model_calls`; they do not
become canonical ATIF steps or contribute to final usage totals. The destination directory must not already exist.

This command does not parse ATOF and does not load NeMo Relay. Each ATIF trajectory takes its agent name from the source
This command does not parse ATIF and does not load NeMo Relay. Each ATIF trajectory takes its agent name from the source
row's `agent_ref.name`; `--agent-version` records the version of that agent implementation.

| Option | Description |
Expand Down Expand Up @@ -850,6 +922,7 @@ Verify rollout quality for an existing run directory. The command reads `<run-di
| `--rollouts-file PATH` | Rollout JSONL path. Relative paths resolve under `RUN_DIR`; absolute paths are used as written. Defaults to `rollouts.jsonl`. |
| `--workers` | Number of worker processes. Defaults to the smaller of the CPU count and 8. |
| `--ignore-checks`, `--ignore` | Comma-separated health-check IDs to exclude from execution and verdict derivation. |
| `--json` | Output health-check results as JSON. |

```bash
gym eval health-check results/my-run
Expand Down Expand Up @@ -939,13 +1012,17 @@ Servers that report `unsupported` or `unknown` cause the command to abort unless
| --- | --- |
| `--config PATH` | Config file for the resources server. Repeatable. |
| `--benchmark NAME` | Load the named benchmark config. |
| `--environment NAME` | Load the named environment config. Repeatable. |
| `--resources-server NAME` | Load the named resources server config. |
| `--model-type NAME` | Load the named model-type config. |
| `--search-dir DIR` | Extra root directory to search for named components. Repeatable. |
| `--input-format gym\|atif` | Select native Gym rollout input or the bounded Relay ATIF v1.7 profile. Defaults to `gym`. |
| `--inputs PATH` | `*_materialized_inputs.jsonl` from `gym eval run`. |
| `--rollouts PATH` | Native Gym `rollouts.jsonl`. Required when `--input-format gym`. |
| `--atif-manifest PATH` | JSONL manifest mapping each Relay ATIF trajectory to `_ng_task_index` and `_ng_rollout_index`, with an optional source SHA-256. Missing hashes produce a warning. Required when `--input-format atif`. |
| `--output PATH`, `-o` | Output JSONL for recomputed rollouts. |
| `--concurrency` | Maximum number of concurrent samples. |
| `--limit` | Maximum number of examples to re-verify. |
| `--force` | Override the `UNSUPPORTED` or `UNKNOWN` reverify mode guard; output is prefixed with `unsafe_`. |
| `--overwrite` | Delete an existing output file instead of raising an error. |
| `--resume` | Resume a partial run: skip rows already in the output and re-verify only the rest. |
Expand Down
Loading