Skip to content

feat: add standalone Gym Docker container - #2161

Merged
kajalj22 merged 32 commits into
mainfrom
add-docker-container
Aug 4, 2026
Merged

kajalj22 merged 32 commits into
mainfrom
add-docker-container

Conversation

@kajalj22

@kajalj22 kajalj22 commented Jul 28, 2026 •

Copy link
Copy Markdown
Contributor

Standalone NeMo-Gym container

FROM cuda-dl-base:26.03 (ubuntu + CUDA, no NGC pytorch)
        │
        ▼
   [base stage]
   Python 3.12, uv, git
        │
        ▼
  [hermetic stage]
  uv sync --extra vllm --locked
  ├── nemo-gym core (ray, fastapi, aiohttp, mlflow …)
  ├── vllm == 0.26.0
  └── flashinfer-python == 0.6.14  (pre-compiled kernels)
        │
        ▼
  [release stage]
  install nemo-gym project
  optional: NEMO_GYM_PREFETCH_SERVERS → per-server .venvs
  entrypoint: gym

Per-server venvs (e.g. local_vllm_model, genrm_model) are separate isolated environments created by gym env start at runtime via uv pip install -e .. Pass NEMO_GYM_PREFETCH_SERVERS to bake them in at build time instead.

CI

build-container.yml runs on pull_request and push to main (when Dockerfile/pyproject.toml/uv.lock change) using ubuntu-latest. Validates the Docker build only — no push, no registry.

Changes

  • docker/Dockerfile — 4-stage build, vllm + flashinfer locked
  • pyproject.toml — [vllm] optional extra (vllm==0.26.0, flashinfer-python==0.6.14)
  • uv.lock — locked packages
  • local_vllm_model/setup.py, genrm_model/setup.py — flashinfer pinned alongside vllm
  • .github/workflows/build-container.yml — build-only CI, triggers on PR and push to main

Dispatch

https://github.com/NVIDIA-NeMo/Gym/actions/workflows/build-container.yml

🤖 Generated with Claude Code

- Add docker/Dockerfile: CUDA base (cuda-dl-base 26.03), Python 3.12
  via uv, Gym core deps via locked uv sync, git pre-installed.
  Supports optional VLLM_VERSION build arg and placeholder
  NEMO_GYM_PREFETCH_CONFIGS arg for future resource-server prefetching.
- Add .github/workflows/build-container.yml: triggers container build
  via the existing _build_container.yml reusable workflow on push/PR
  to main when Dockerfile, pyproject.toml, or uv.lock change.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@kajalj22
kajalj22 requested a review from a team as a code owner July 28, 2026 01:13
@kajalj22
kajalj22 marked this pull request as draft July 28, 2026 01:16
@copy-pr-bot

copy-pr-bot Bot commented Jul 28, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

NVIDIA self-hosted runners (linux-amd64-cpu16) block workflows
triggered by pull_request events per org security policy. Container
builds only run on push to main and workflow_dispatch.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@kajalj22

Copy link
Copy Markdown
Contributor Author

/claude review

Comment thread docker/Dockerfile Resolved
Comment thread docker/Dockerfile Resolved
kajalj22 and others added 3 commits July 27, 2026 21:23
- Add ENV VIRTUAL_ENV=/opt/nemo_gym_venv in hermetic stage so uv pip
  install (used for optional vLLM) correctly targets the project venv.
  uv pip ignores UV_PROJECT_ENVIRONMENT; it needs VIRTUAL_ENV or --python.
- Remove NEMO_GYM_PREFETCH_CONFIGS arg and dead gym env prefetch block.
  The prefetch CLI subcommand doesn't exist yet; passing the arg would
  fail the build. Will re-add once the command is implemented in Gym.
- Add --link-mode symlink to hermetic uv sync, matching RL's pattern.
- Remove duplicate WORKDIR in release stage.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
vLLM version should be locked via pyproject.toml + uv.lock and installed
with uv sync --extra vllm --locked, not pinned by the caller at build time.
Remove the VLLM_VERSION ARG and uv pip install block entirely. Will re-add
as a proper uv extra once the vllm optional dependency is wired in pyproject.toml.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
- pyproject.toml: add [vllm] optional-dependencies extra with vllm as dep
- uv.lock: resolve vllm==0.26.0 and its transitive deps (293 packages total)
- docker/Dockerfile: install --extra vllm in both the hermetic dep-sync
  layer and the release project-install layer so vLLM is baked in and
  version-locked via the lockfile (not caller-controlled)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@kajalj22

Copy link
Copy Markdown
Contributor Author

/claude review

Gym uses static versioning (attr = "nemo_gym.__version__"), not
setuptools-scm, so git tag history is not needed at build time.
The unshallow step and its comment were copied verbatim from RL
where setuptools-scm is used. Remove both.

Also remove the redundant WORKDIR /opt/nemo-gym at the end of
the release stage — it is inherited from hermetic and already set
before the uv sync runs.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Without --build-context nemo-gym=<path>, the hermetic stage fetches
pyproject.toml and uv.lock from GitHub main (via ADD --keep-git-dir),
missing changes on the PR branch (e.g. the vllm optional extra).

FW CI templates check out the repo at path: github.run_id, so the
correct build context is nemo-gym=<run_id>/ relative to the workspace.
Add build-contexts input to _build_container.yml and forward it through.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Comment thread docker/Dockerfile Resolved
Comment thread pyproject.toml Resolved
Comment thread docker/Dockerfile Resolved
Comment thread docker/Dockerfile Resolved
vLLM version sync:
- pyproject.toml [vllm] extra: vllm==0.26.0 + flashinfer-python==0.6.14
  (exact version vllm 0.26.0 specifies; ensures pre-compiled kernels are
  used to avoid JIT compilation on first generation / slow step time)
- responses_api_models/local_vllm_model/setup.py: 0.20.0 → 0.26.0
- responses_api_models/genrm_model/setup.py: 0.20.0 → 0.26.0
  (must stay in sync per the comment in genrm setup.py)
- uv.lock: re-locked, 293 packages (flashinfer-python 0.6.14 was already
  a transitive dep of vllm 0.26.0, no net change)

Prefetch:
- docker/Dockerfile: add NEMO_GYM_PREFETCH_SERVERS build arg
  Pass a space-separated list of server paths relative to the repo root.
  For each server, creates its isolated .venv via uv sync at build time,
  so the venv is ready at runtime without any download overhead.
  Example: --build-arg NEMO_GYM_PREFETCH_SERVERS="responses_api_models/local_vllm_model"

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
kajalj22 and others added 2 commits July 28, 2026 14:15
uv sync reads [project.dependencies] from pyproject.toml but server deps
(vllm, uvicorn, hf_transfer, etc.) are declared in setup.py install_requires
under dynamic = ["dependencies"] — uv sync never sees them and creates an
empty venv.

Fix mirrors gym env start (setup_command.py:141):
  uv pip install -e .   runs setup.py and picks up install_requires correctly
  --python .venv/bin/python   targets the server .venv, not UV_PROJECT_ENVIRONMENT
                              which is globally set to /opt/nemo_gym_venv

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
After a successful build, the 'test' job pulls the container from ACR
and runs two checks:
- gym --help: verifies the CLI entrypoint works
- import check: asserts vllm==0.26.0 and flashinfer-python==0.6.14 are
  installed at the expected versions (uses importlib.metadata to avoid
  triggering CUDA initialisation on a CPU runner)

Also expose container-uri output from _build_container.yml so
build-container.yml can reference the built image URI.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
kajalj22 and others added 2 commits July 28, 2026 14:58
ENTRYPOINT is ["gym"] so 'docker run <img> gym --help' invokes
'gym gym --help' which fails. Correct form is 'docker run <img> --help'.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Adds flashinfer-python at the exact version vllm 0.26.0 requires,
matching the pyproject.toml [vllm] extra. Ensures per-server venvs
use pre-compiled CUDA kernels and avoid JIT compilation on first
generation. Mac path unchanged (vllm==0.11.0, no flashinfer).

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
kajalj22 and others added 2 commits July 29, 2026 22:42
Switch to ubuntu-latest (GitHub-hosted) so the build runs during PRs
as a required check. Self-hosted runners block pull_request events, so
ubuntu-latest is needed for PR validation.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@kajalj22
kajalj22 requested a review from chtruong814 July 30, 2026 04:34
chtruong814
chtruong814 previously approved these changes Jul 30, 2026
ananthsub
ananthsub previously approved these changes Jul 30, 2026
Comment thread docker/Dockerfile Resolved
@github-actions github-actions Bot removed the sla:review-overdue Review response is over the one-business-day SLA label Jul 30, 2026
kajalj22 and others added 3 commits July 30, 2026 18:59
Per review feedback (Ananth), prefetch should work like RL's
prefetch_venvs.py — accepting Gym configs and running a dry
install of deps. Removing the current incomplete implementation
to unblock shipping the container; proper prefetch via
'gym env prefetch' will land in a follow-up PR.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Restores the path-based prefetch build arg removed in the previous commit.
A follow-up PR (branching from this one) will replace this with a
config-based 'gym env prefetch' command similar to RL's prefetch_venvs.py.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
kajalj22 added a commit that referenced this pull request Aug 3, 2026
Placeholder for 'gym env prefetch' implementation.
Addresses review feedback from #2161.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@kajalj22 kajalj22 added the r0.5.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge. label Aug 3, 2026
kajalj22 and others added 3 commits August 3, 2026 20:21
…#2255)

## Summary

Follow-up to #2161 (standalone Gym container). Implements `gym env
prefetch` — a proper config-driven CLI command for pre-warming server
venvs at container build time, similar to how RL's `prefetch_venvs.py`
works.

## What this replaces

PR #2161 has `NEMO_GYM_PREFETCH_SERVERS` which takes space-separated
directory paths. This PR replaces it with `NEMO_GYM_PREFETCH_CONFIGS`
which takes Gym config files — the same format as `gym env start`. This
is the approach Ananth described in the review.

## How it works

```
gym env prefetch "+config_paths=[config1.yaml,config2.yaml]"
```

1. Parses configs the same way `gym env start` does
2. For each server found in the config, calls `setup_env_command()` —
creates the venv and installs deps serially
3. Exits — **no Ray initialised, no server processes started**

Serial installation reduces uv cache size (same reasoning as `dry_run`
mode in `RunHelper.start`).

## Dockerfile usage

```dockerfile
# Pre-warm venvs for your server configs at build time:
docker buildx build \
  --build-arg NEMO_GYM_PREFETCH_CONFIGS="responses_api_models/local_vllm_model/configs/local_vllm_model.yaml,resources_servers/math/configs/math.yaml" \
  -f docker/Dockerfile .
```

## Changes

- `nemo_gym/cli/env.py` — `gym env prefetch` command
- `docker/Dockerfile` — `NEMO_GYM_PREFETCH_CONFIGS` replaces
`NEMO_GYM_PREFETCH_SERVERS`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@kajalj22
kajalj22 enabled auto-merge (squash) August 4, 2026 19:08
@kajalj22
kajalj22 disabled auto-merge August 4, 2026 22:33
@kajalj22
kajalj22 merged commit e590c4f into main Aug 4, 2026
16 of 17 checks passed
@kajalj22
kajalj22 deleted the add-docker-container branch August 4, 2026 22:33
kajalj22 added a commit that referenced this pull request Aug 5, 2026
…2334)

beep boop [🤖]: Hi @kajalj22 👋,

    we've cherry picked #2161 into  for you! 🚀

    Please review and approve this cherry pick by your convenience!

Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: kajalj22 <kajalj@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
OlegSudakov pushed a commit to OlegSudakov/Gym that referenced this pull request Aug 7, 2026
## Standalone NeMo-Gym container

```
FROM cuda-dl-base:26.03 (ubuntu + CUDA, no NGC pytorch)
        │
        ▼
   [base stage]
   Python 3.12, uv, git
        │
        ▼
  [hermetic stage]
  uv sync --extra vllm --locked
  ├── nemo-gym core (ray, fastapi, aiohttp, mlflow …)
  ├── vllm == 0.26.0
  └── flashinfer-python == 0.6.14  (pre-compiled kernels)
        │
        ▼
  [release stage]
  install nemo-gym project
  optional: NEMO_GYM_PREFETCH_SERVERS → per-server .venvs
  entrypoint: gym
```

**Per-server venvs** (e.g. `local_vllm_model`, `genrm_model`) are
separate isolated environments created by `gym env start` at runtime via
`uv pip install -e .`. Pass `NEMO_GYM_PREFETCH_SERVERS` to bake them in
at build time instead.

## CI

`build-container.yml` runs on `pull_request` and `push` to main (when
`Dockerfile`/`pyproject.toml`/`uv.lock` change) using `ubuntu-latest`.
Validates the Docker build only — no push, no registry.

## Changes

- `docker/Dockerfile` — 4-stage build, vllm + flashinfer locked
- `pyproject.toml` — `[vllm]` optional extra (`vllm==0.26.0`,
`flashinfer-python==0.6.14`)
- `uv.lock` — locked packages
- `local_vllm_model/setup.py`, `genrm_model/setup.py` — flashinfer
pinned alongside vllm
- `.github/workflows/build-container.yml` — build-only CI, triggers on
PR and push to main

## Dispatch

https://github.com/NVIDIA-NeMo/Gym/actions/workflows/build-container.yml

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
linj-glitch pushed a commit to linj-glitch/Gym that referenced this pull request Oct 15, 2026
…DIA-NeMo#4639)

## What does this PR do?

Three pages under `fern/versions/latest/pages` cite
`resources_servers/*/configs/*.yaml` files that do not exist on main.
This PR points each one at the file that replaced it. It also adds a
docs test so the next rename that leaves a citation behind fails CI.

No issue or NVBug: found by scanning the latest pages for cited config
paths and checking each one exists:

```bash
grep -rhoE 'resources_servers/[a-z0-9_]+/configs/[A-Za-z0-9_.-]+\.yaml' fern/versions/latest/pages \
  | sort -u | while read f; do [ -f "$f" ] || echo MISSING $f; done
```

| Page | Cited path | Now | Why |
| --- | --- | --- | --- |
| `reference/cli-commands.mdx` (`gym env prefetch` example) |
`resources_servers/math/configs/math.yaml` |
`resources_servers/math_with_judge/configs/math_with_judge.yaml` |
`resources_servers/math/` never existed. The example came from the
`prefetch` docstring added in NVIDIA-NeMo#2161 and was copied into the page by
NVIDIA-NeMo#4093. `math_with_judge.yaml` is the math config the same page already
uses for `ng_e2e_collect_rollouts`. The docstring in
`nemo_gym/cli/env.py` gets the same fix. |
|
`contribute/rl-framework-integration/gym-rl-framework-integration-success-criteria.mdx`
| `math_with_judge/configs/bytedtsinghua_dapo17k.yaml` (pinned to
`299e8c04`) | `math_with_judge/configs/dapo17k.yaml` on `main` | NVIDIA-NeMo#859
deleted `bytedtsinghua_dapo17k.yaml` and moved its DAPO17k train and
AIME24 validation sets into `dapo17k.yaml`. The pinned permalink still
resolves, but it points at a config that main no longer ships. The other
links in this page's tables already use `main`. |
| `reference/faq.mdx` (4 dataset upload/migrate examples) |
`resources_servers/multineedle/configs/multineedle.yaml` |
`resources_servers/example_multi_step/configs/example_multi_step.yaml` |
NVIDIA-NeMo#291 renamed `multineedle` to `example_multi_step`. The upload command
reads `domain` from this config, and `example_multi_step.yaml` sets it.
|

Versioned snapshots under `fern/versions/v*/` are frozen and left as
they shipped.

### Test


`tests/unit_tests/test_fern_docs_links.py::test_cited_resources_server_configs_exist`
scans every latest page for
`resources_servers/<name>/configs/<file>.yaml` and asserts that each
file exists. It skips `my_weather_tool`, which
`environment-tutorials/single-step-environment.mdx` has the reader
create. It also asserts that it found at least one citation, so a broken
regex can't make it pass vacuously.

## Before the fix

Gym `974d1a921` (main). The new test fails (`1 failed`) and reports
these six citations:

```
fern/versions/latest/pages/contribute/rl-framework-integration/gym-rl-framework-integration-success-criteria.mdx:38: resources_servers/math_with_judge/configs/bytedtsinghua_dapo17k.yaml
fern/versions/latest/pages/reference/cli-commands.mdx:695: resources_servers/math/configs/math.yaml
fern/versions/latest/pages/reference/faq.mdx:41: resources_servers/multineedle/configs/multineedle.yaml
fern/versions/latest/pages/reference/faq.mdx:53: resources_servers/multineedle/configs/multineedle.yaml
fern/versions/latest/pages/reference/faq.mdx:91: resources_servers/multineedle/configs/multineedle.yaml
fern/versions/latest/pages/reference/faq.mdx:108: resources_servers/multineedle/configs/multineedle.yaml
```

The documented `gym env prefetch` example, run with venv installs
stubbed out so that only config resolution and server discovery run
(`policy_model_name` is normally supplied by `env.yaml`):

```
$ gym env prefetch "+config_paths=[responses_api_models/local_vllm_model/configs/local_vllm_model.yaml,resources_servers/math/configs/math.yaml]" ++policy_model_name=Qwen/Qwen3-4B-Instruct-2507
Error: config_paths entry 'resources_servers/math/configs/math.yaml' was not found. Looked in:
  - <Gym>/resources_servers/math/configs/math.yaml
Check the path is spelled correctly and is relative to your working directory, an extra root
(NEMO_GYM_EXTRA_ROOTS / --search-dir), or the Gym install root.
exit=1
```

## After the fix

```
$ gym env prefetch "+config_paths=[responses_api_models/local_vllm_model/configs/local_vllm_model.yaml,resources_servers/math_with_judge/configs/math_with_judge.yaml]" ++policy_model_name=Qwen/Qwen3-4B-Instruct-2507
Pre-warming venv for: responses_api_models/local_vllm_model
Pre-warming venv for: resources_servers/math_with_judge
Pre-warming venv for: responses_api_agents/simple_agent
Pre-warming venv for: environment_servers/legacy_agent
Prefetch complete.
exit=0
```

`gym env resolve --config` now finds `dapo17k.yaml` and
`example_multi_step.yaml`. It then stops at the usual
unset-required-values check (`policy_model` / judge name), the same as
any config resolved without an `env.yaml`.

## Validation

- `pytest tests/unit_tests/test_fern_docs_links.py
tests/unit_tests/test_opensandbox_documented_config.py`: 25 passed, 21
subtests passed.
- `pytest tests/unit_tests/test_cli_main.py`: passed (the `env.py`
change only touches a docstring).
- `pre-commit run --all-files`: all hooks pass.
- Rollouts: N/A. The only change outside docs and tests is the example
text in one docstring.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: qiaochuz <qiaochuz@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

r0.5.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants