docs: shorten benchmark onboarding guidance by ritaneves · Pull Request #3713 · NVIDIA-NeMo/Gym

ritaneves · 2026-09-28T14:45:15Z

What does this PR do?

Shorten benchmark onboarding into scaffold, prepare, validate/test, smoke, and local publish steps. Clarify manifest prompting and verifier reuse, and distinguish current Environment Server/legacy adapter routing from native episode-protocol migration (#1866, #3724).

Validation

Validated at 6eb93be01:

pre-commit run --all-files — passed.
python tests/unit_tests/test_fern_docs_links.py — 18 passed.
python -m pytest --import-mode=importlib tests/unit_tests/test_global_config.py -k config_without_environment_server_is_rejected -o addopts='' -q — 1 passed.
cd fern && npm run check -- --warnings — 0 errors; authenticated redirects check skipped because Fern login is unavailable.
git diff --check — passed.

Rollouts

N/A: documentation-only changes; no runtime, agent, or verifier behavior changes.

Compatibility and benchmark impact

No runtime or benchmark-result changes. The guide now describes the existing requirement that runnable agents have Environment Server routing; compatibility routing is distinct from migration to native protocols.

Checklist

I have read the contributing guidelines.
The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
Documentation added or updated, or N/A with justification.
Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
New source files include the required Apache-2.0 SPDX header.
All commits have DCO sign-off (git commit -s) (so the DCO check passes).

copy-pr-bot · 2026-09-28T14:45:19Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-09-28T14:47:50Z

🌿 Preview your docs: https://nvidia-preview-rfneves-gym-benchmark-onboarding-docs.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

ritaneves · 2026-09-28T15:21:25Z

/claude review

claude · 2026-09-28T15:24:16Z

SHIP — LGTM, no reliability concerns.

Docs-only change to two .mdx files under fern/versions/latest/pages/contribute/environments/. No code, config, or verifier logic touched, so no compute or rollout evidence needed.

Verified against the trusted base source:

Manifest fields documented match nemo_gym/environment/manifest.py: prompt_source (template/prepared/agent with the stated standard_prompt_config rules), config_path (defaults to config.yaml, must be relative and stay inside the catalog tree), dataset_owner, and the reward.range change to lower <= upper (constant range allowed) all reflect the real Pydantic model and validators.
resources_server: null claim is correct: field is nullable and required except for custom-gym-verifier (manifest.py lines 88, 249).
Internal links: the added /main/ prefixes are the correct published-URL convention for latest/pages/ content (per fern/README.md; 10 other pages already use it). All link targets exist (new-environment, development-setup, single-step-environment, model-server, about/architecture, adding-a-benchmark) and all cross-page anchors resolve (#component-ownership, #environment-manifest, #integration-profiles, #prompt-sources).

No findings.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33

Fixed the outdated architecture wording in 6eb93be: Environment Server routing is already required; legacy forwarding and native protocol migration are now distinguished. Docs/link checks and pre-commit pass (authenticated redirect checking unavailable). LGTM.

yaoyu-33 · 2026-10-02T00:48:00Z

/ok to test 6eb93be

github-actions · 2026-10-02T15:57:42Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/37030616925

github-actions · 2026-10-02T16:01:31Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/37031058893

sephmard · 2026-10-02T18:49:07Z

+| `data/source.jsonl`, `data/example.jsonl` | Replace the sample task and retain representative examples |
+| `README.md` | Measurement contract, preparation and evaluation commands, and limitations |
+
+It also creates `resources_servers/my_benchmark/` with a verifier implementation, config, dependencies, unit tests, and verifier fixture. Reuse a harness for a new task; a new dataset or prompt does not by itself require a new Agent Server. See [Single Step Environment](/main/environment-tutorials/single-step-environment) when implementing new task tools or scoring.


We'll need to make a note to update this as we are defining the task definition

sephmard · 2026-10-02T19:03:19Z

-The most important principle when adding benchmarks into NeMo Gym is ensuring the fidelity of the benchmark. As a result, there are additional steps and best practices to adding a benchmark that are required on top of adding just a training environment (although the steps below are still suggested for training environments).

-## Reward profiling
+# Add a benchmark


It might be good to have a "hello world" example. The detail below is excellent. But it might be good to show an overview so users are not intimidated by the full detail, and they percieve the onboarding to be straight forward.

This can be a follow up, but do think its important.

Flow would be

Overview of steps 1-5

Hello world basic example

Step 1

...

Step 5

…IA-NeMo#4043) ## What does this PR do? Follow up on the review of NVIDIA-NeMo#3713 by introducing benchmark onboarding with a five-step overview and a concrete hello-world example before the detailed instructions. The example uses the generated one-question scaffold, explains its prompt and exact-match reward, and shows an optional one-row model evaluation. Make local publication an explicit fifth step. Add task-contract maintenance guidance and a maintainer reminder to update the scaffold file/responsibility mapping as the shared task definition evolves. ## Validation - `pre-commit run --all-files` — passed. - `python tests/unit_tests/test_fern_docs_links.py` — 18 tests and 17 subtests passed. - `fern check --warnings` — 0 errors; authenticated redirect checking skipped because Fern login is unavailable. - Generated scaffold checked through the Python API in a temporary directory: preparation output, prompt rendering without the reference answer, manifest parsing, and correct/incorrect/malformed verifier fixture cases. - `git diff --check` — passed. Full CLI execution was not verified because the existing local virtual environment lacks `httptools`. No model evaluation was run. ## Compatibility and benchmark impact Documentation only; no runtime, verifier, or benchmark-result changes. Frozen release documentation is unchanged. The commit includes DCO sign-off. --------- Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com> Signed-off-by: ritaneves <rfernandesne@nvidia.com>

ritaneves temporarily deployed to public September 28, 2026 15:09 — with GitHub Actions Inactive

ritaneves temporarily deployed to public September 28, 2026 15:11 — with GitHub Actions Inactive

ritaneves temporarily deployed to public September 28, 2026 15:21 — with GitHub Actions Inactive

ritaneves added this pull request to stack #3710 September 28, 2026 15:21

ritaneves temporarily deployed to public September 28, 2026 15:23 — with GitHub Actions Inactive

Glorf previously approved these changes Sep 29, 2026

View reviewed changes

ritaneves dismissed Glorf’s stale review via 8cc96d9 September 29, 2026 12:25

ritaneves force-pushed the rfneves/gym-benchmark-onboarding-docs branch from 5b77680 to 8cc96d9 Compare September 29, 2026 12:25

ritaneves temporarily deployed to public September 29, 2026 12:26 — with GitHub Actions Inactive

nemo-automation-bot Bot requested a review from a team September 29, 2026 12:26

copy-pr-bot Bot temporarily deployed to public September 29, 2026 12:26 Inactive

ritaneves temporarily deployed to public September 29, 2026 12:27 — with GitHub Actions Inactive

ritaneves temporarily deployed to public September 29, 2026 12:28 — with GitHub Actions Inactive

copy-pr-bot Bot temporarily deployed to public September 29, 2026 12:28 Inactive

ritaneves temporarily deployed to public September 29, 2026 12:28 — with GitHub Actions Inactive

copy-pr-bot Bot temporarily deployed to public September 29, 2026 12:29 Inactive

ritaneves temporarily deployed to public September 29, 2026 14:48 — with GitHub Actions Inactive

ritaneves temporarily deployed to public September 29, 2026 14:49 — with GitHub Actions Inactive

ritaneves requested a review from sephmard September 29, 2026 14:54

github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Sep 30, 2026

github-actions Bot mentioned this pull request Sep 30, 2026

[Tracker] Pull request handoff SLA #3402

Open

docs: clarify current environment server routing

6eb93be

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

yaoyu-33 temporarily deployed to public September 30, 2026 17:46 — with GitHub Actions Inactive

yaoyu-33 approved these changes Sep 30, 2026

View reviewed changes

yaoyu-33 added ready-to-merge PR is approved, current, and waiting for final CI or merge and removed needs-review PR is ready for code review and waiting on a reviewer labels Sep 30, 2026

yaoyu-33 temporarily deployed to public September 30, 2026 17:48 — with GitHub Actions Inactive

yaoyu-33 temporarily deployed to public September 30, 2026 17:49 — with GitHub Actions Inactive

copy-pr-bot Bot temporarily deployed to public October 2, 2026 00:48 Inactive

copy-pr-bot Bot temporarily deployed to public October 2, 2026 00:50 Inactive

copy-pr-bot Bot temporarily deployed to public October 2, 2026 00:51 Inactive

yuhezhang-ai added this pull request to the merge queue Oct 2, 2026

Merged via the queue into main with commit bf04ab6 Oct 2, 2026
57 checks passed

yuhezhang-ai deleted the rfneves/gym-benchmark-onboarding-docs branch October 2, 2026 16:03

sephmard reviewed Oct 2, 2026

View reviewed changes

sephmard mentioned this pull request Oct 2, 2026

docs(contribute): add Overview, Hello World, and Reward Profiling to Add a Benchmark #3956

Closed

4 tasks done

ritaneves mentioned this pull request Oct 5, 2026

docs: add benchmark onboarding overview and hello world example #4043

Merged

Navigation Menu

Repository navigation

docs: shorten benchmark onboarding guidance - #3713

docs: shorten benchmark onboarding guidance#3713
yuhezhang-ai merged 5 commits into
mainfrom
rfneves/gym-benchmark-onboarding-docs

ritaneves commented Sep 28, 2026 •

edited by yaoyu-33

Loading

Uh oh!

copy-pr-bot Bot commented Sep 28, 2026

Uh oh!

github-actions Bot commented Sep 28, 2026

Uh oh!

ritaneves commented Sep 28, 2026

Uh oh!

claude Bot commented Sep 28, 2026

Uh oh!

yaoyu-33 left a comment

Uh oh!

yaoyu-33 commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

Uh oh!

sephmard Oct 2, 2026

Uh oh!

sephmard Oct 2, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

ritaneves commented Sep 28, 2026 • edited by yaoyu-33 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Validation

Rollouts

Compatibility and benchmark impact

Checklist

Uh oh!

copy-pr-bot Bot commented Sep 28, 2026

Uh oh!

github-actions Bot commented Sep 28, 2026

Uh oh!

ritaneves commented Sep 28, 2026

Uh oh!

claude Bot commented Sep 28, 2026

Uh oh!

yaoyu-33 left a comment

Choose a reason for hiding this comment

Uh oh!

yaoyu-33 commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

Uh oh!

sephmard Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

sephmard Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ritaneves commented Sep 28, 2026 •

edited by yaoyu-33

Loading