Repository navigation
Gym 0.2.0 huggingface dataset pointers - #859
Merged
Merged
Conversation
…p configs Signed-off-by: Frankie Siino <fsiino@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
Signed-off-by: Frankie Siino <fsiino@nvidia.com>
…nters-0.2.0 Signed-off-by: Frankie Siino <fsiino@nvidia.com> # Conflicts: # README.md
…nters-0.2.0 Signed-off-by: Frankie Siino <fsiino@nvidia.com>
fsiino-nvidia
marked this pull request as ready for review
March 10, 2026 23:01
Signed-off-by: Brian Yu <bxyu@nvidia.com>
Signed-off-by: Brian Yu <bxyu@nvidia.com>
bxyu-nvidia
added a commit
that referenced
this pull request
Mar 11, 2026
This adds hf pointers and descriptions for the new datasets with super release into the configs + docs. --------- Signed-off-by: Frankie Siino <fsiino@nvidia.com> Signed-off-by: Brian Yu <bxyu@nvidia.com> Co-authored-by: bxyu-nvidia <bxyu@nvidia.com> Signed-off-by: Brian Yu <bxyu@nvidia.com>
raveedturing
pushed a commit
to turing-rlgym/Gym
that referenced
this pull request
Mar 18, 2026
This adds hf pointers and descriptions for the new datasets with super release into the configs + docs. --------- Signed-off-by: Frankie Siino <fsiino@nvidia.com> Signed-off-by: Brian Yu <bxyu@nvidia.com> Co-authored-by: bxyu-nvidia <bxyu@nvidia.com>
mahanfathi
pushed a commit
that referenced
this pull request
Mar 24, 2026
This adds hf pointers and descriptions for the new datasets with super release into the configs + docs. --------- Signed-off-by: Frankie Siino <fsiino@nvidia.com> Signed-off-by: Brian Yu <bxyu@nvidia.com> Co-authored-by: bxyu-nvidia <bxyu@nvidia.com>
This was referenced Oct 12, 2026
khazic
pushed a commit
to khazic/Gym-lao
that referenced
this pull request
Oct 15, 2026
…EADME (NVIDIA-NeMo#4574) ## What does this PR do? The math_with_judge README's "Prepare for trajectory collection" command works again. It loads `configs/dapo17k_trajectory_collection.yaml`, which no longer exists, so `gym dataset collate` exited with `ConfigPathNotFoundError` before downloading anything. This PR restores that config in the current schema, documents the step's prerequisites, and adds a regression test that replays the README command. Running the README's other GitLab download, OpenMathReasoning, from outside the Gym root exposed a second bug: the GitLab downloader did not create the output directory. The second commit fixes it. Fixes NVBug [6847517](https://nvbugspro.nvidia.com/bug/6847517). Closes NVIDIA-NeMo#4498. ### Cause NVIDIA-NeMo#859 ("Gym 0.2.0 huggingface dataset pointers") deleted three dapo17k variants under `resources_servers/math_with_judge/configs/`, including `dapo17k_trajectory_collection.yaml`, and removed their root-README rows. The math_with_judge README, which has cited that file since NVIDIA-NeMo#291, was not updated. Config resolution (`nemo_gym/global_config.py` `load_extra_config_paths`) then raises `ConfigPathNotFoundError`. `dapo17k.yaml` is not a drop-in replacement. The deleted config put the DAPO17k train problems in the **validation** split, repeated 16 times, because NeMo RL with `is_trajectory_collection: true` collects from the validation dataloader only. `dapo17k.yaml` has DAPO17k as `train` ×1 and AIME24 ×32 as `validation`, so trajectory collection would run on AIME24. Restoring the old file verbatim would not work either. It set `judge_model_server.name: ???`, and since NVIDIA-NeMo#1575 the parser rejects any `???` left after merging (`ConfigMissingValuesError`). The second bug is in `download_jsonl_dataset` (`nemo_gym/gitlab_utils.py`). It opened `output_fpath` for writing without creating its directory. A Gym checkout tracks `data/`, so the README's `--output data/open_math_reasoning_problems.jsonl` works from the Gym root but fails with `FileNotFoundError` from any other directory. `gym dataset collate --download` fails the same way for GitLab-backed datasets. The Hugging Face download already creates the directory (`hf_utils.py`). ### Fix - Restore `configs/dapo17k_trajectory_collection.yaml`. The validation split is the `dapo17k` 0.0.1 `dapo17k_train.jsonl` artifact (the same GitLab source as `dapo17k.yaml`), `num_repeats: 16`. Grading uses math-verify only (`should_use_judge: false`), so `judge_model_server.name` is `policy_model` and the README command needs no override. The file has `description`/`value` and the environment-server block like `dapo17k.yaml`. The `update-readme-table` hook re-adds its root-README row. - It keeps its own local file, `data/dapo17k_trajectory_collection_train.jsonl`, as before NVIDIA-NeMo#859. Collate checks each dataset against `<jsonl stem>_metrics.json`. Sharing `dapo17k.yaml`'s `dapo17k_train.jsonl` would compare the ×16 validation stats with that file's train sidecar and fail (`train != validation`, `17398 != 278368`). - Add `data/dapo17k_trajectory_collection_train_metrics.json`, written by a real collate run (278368 rows = 17398 × 16). Its word and turn statistics match `dapo17k_train_metrics.json` exactly, so the rows are the same. - README: the command is unchanged. One note says the GitLab downloads in this README (the OpenMathReasoning download and this `--download`) need the `mlflow` extra (optional since NVIDIA-NeMo#3547) and `mlflow_tracking_uri` / `mlflow_tracking_token` in `env.yaml`. The section says what it writes and why the split is validation. Test: `tests/unit_tests/test_math_with_judge_readme_trajectory_config.py` takes the README's `gym dataset collate` block and runs it through the real CLI with `dispatch` intercepted. It resolves the config offline (no `env.yaml`), checks the single dataset is `validation` ×16 with `dapo17k.yaml`'s GitLab identifier, and checks the committed sidecar describes that split. It then runs `TrainDataProcessor` on a three-row fixture in a temporary directory, without network access, and checks that only `validation.jsonl` and its metrics come out, with each question repeated 16 times in order. - `download_jsonl_dataset` now creates the output's parent directory, as the Hugging Face download does. A new test in `tests/unit_tests/test_gitlab_utils.py` fakes the registry lookup and the HTTP fetch, then downloads into a missing nested directory. It fails before this change with the same `FileNotFoundError` and passes after. These plausible wrong fixes all fail the trajectory test: - restoring the file verbatim (`???`); - pointing the README at `dapo17k.yaml` with a judge override; - `num_repeats: 1`; - a `train` split; - sharing `dapo17k_train.jsonl`. ### Before the fix Gym `977583d3d` (`main`), the README command from the Gym root: ```bash gym dataset collate \ --config resources_servers/math_with_judge/configs/dapo17k_trajectory_collection.yaml \ --config responses_api_models/openai_model/configs/openai_model.yaml \ --output-dir data/dapo17k_trajectory_collection \ --mode train_preparation \ --download ``` ```text nemo_gym.config_types.ConfigPathNotFoundError: config_paths entry 'resources_servers/math_with_judge/configs/dapo17k_trajectory_collection.yaml' was not found. Looked in: - <gym>/resources_servers/math_with_judge/configs/dapo17k_trajectory_collection.yaml Check the path is spelled correctly and is relative to your working directory, an extra root (NEMO_GYM_EXTRA_ROOTS / --search-dir), or the Gym install root. EXIT=1 ``` The README's OpenMathReasoning GitLab download, from a directory without `data/`, same base: ```bash gym dataset download --storage gitlab --name math_open_math_reasoning --revision 0.0.1 \ --artifact open_math_reasoning_problems.jsonl --output data/open_math_reasoning_problems.jsonl ``` ```text File ".../nemo_gym/gitlab_utils.py", line 111, in download_jsonl_dataset with open(config.output_fpath, "w") as f: FileNotFoundError: [Errno 2] No such file or directory: '<workspace>/data/open_math_reasoning_problems.jsonl' ``` ### After the fix This branch, same command, with the `mlflow` extra and GitLab registry credentials in `env.yaml`. No local data beforehand: ```text Downloading dataset `validation` for `math_with_judge_simple_agent` from gitlab using dataset_name='dapo17k' version='0.0.1' artifact_fpath='dapo17k_train.jsonl' output_fpath='resources_servers/math_with_judge/data/dapo17k_trajectory_collection_train.jsonl' Dataset validation repeating 16x: each line repeated 16 times (e.g. pattern: abc -> aaaabbbbcccc) Aggregate metrics for resources_servers/math_with_judge/data/dapo17k_trajectory_collection_train_metrics.json Aggregate metrics for data/dapo17k_trajectory_collection/validation_metrics.json EXIT=0 $ wc -l data/dapo17k_trajectory_collection/validation.jsonl 278368 data/dapo17k_trajectory_collection/validation.jsonl ``` There are 17398 runs of identical questions, each 16 rows long, and no `train.jsonl`. A second run skips the download, reports the sidecar `already up to date`, and exits 0. The OpenMathReasoning download, same command and directory: exit 0, with the file written (the QA case below checks its rows). ### Validation - QA testcase `math_with_judge`, which runs both README GitLab steps from a scratch directory with registry credentials: - base `977583d3d`: 0/2. Trajectory collation fails because the config is missing; the download fails with `FileNotFoundError`. - this branch: 2/2. The download passes in 4.4 s; the trajectory collation passes in 20.3 s with 17398 × 16 rows and the committed sidecar unchanged. - The QA-suite change ships separately. - New tests: trajectory test before 1 failed (`ConfigPathNotFoundError`), after 1 passed, and the five wrong fixes above fail it. Downloader test before 1 failed (`FileNotFoundError`), after 1 passed, also with the real `mlflow` 3.16.1 installed. - `test_train_data_utils.py`, `test_cli_main.py`, `test_global_config.py`, `test_dataset_source.py`, `test_task_data.py::TestSchemaPresence`, `test_ray_opt_out.py`, `test_cli_resources_servers.py`, `test_model_registry.py`, `test_resources_server_registry.py`, `test_environment_scaffold.py`, `test_gitlab_utils.py`, `test_hf_utils.py`, and the new tests: 592 passed. - `pre-commit run --all-files`: passed. - Rollout evidence: N/A. This is a config and docs change, and `gym dataset collate` runs no model. The full download-and-collate run above uses the real registry data. ### Compatibility A new config file. Nothing that loads other configs changes. GitLab downloads now create a missing output directory instead of failing. The restored config uses the same GitLab artifact as `dapo17k.yaml`, but a separate local file, so each keeps its own metrics sidecar. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: qiaochuz <qiaochuz@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
linj-glitch
pushed a commit
to linj-glitch/Gym
that referenced
this pull request
Oct 15, 2026
…DIA-NeMo#4639) ## What does this PR do? Three pages under `fern/versions/latest/pages` cite `resources_servers/*/configs/*.yaml` files that do not exist on main. This PR points each one at the file that replaced it. It also adds a docs test so the next rename that leaves a citation behind fails CI. No issue or NVBug: found by scanning the latest pages for cited config paths and checking each one exists: ```bash grep -rhoE 'resources_servers/[a-z0-9_]+/configs/[A-Za-z0-9_.-]+\.yaml' fern/versions/latest/pages \ | sort -u | while read f; do [ -f "$f" ] || echo MISSING $f; done ``` | Page | Cited path | Now | Why | | --- | --- | --- | --- | | `reference/cli-commands.mdx` (`gym env prefetch` example) | `resources_servers/math/configs/math.yaml` | `resources_servers/math_with_judge/configs/math_with_judge.yaml` | `resources_servers/math/` never existed. The example came from the `prefetch` docstring added in NVIDIA-NeMo#2161 and was copied into the page by NVIDIA-NeMo#4093. `math_with_judge.yaml` is the math config the same page already uses for `ng_e2e_collect_rollouts`. The docstring in `nemo_gym/cli/env.py` gets the same fix. | | `contribute/rl-framework-integration/gym-rl-framework-integration-success-criteria.mdx` | `math_with_judge/configs/bytedtsinghua_dapo17k.yaml` (pinned to `299e8c04`) | `math_with_judge/configs/dapo17k.yaml` on `main` | NVIDIA-NeMo#859 deleted `bytedtsinghua_dapo17k.yaml` and moved its DAPO17k train and AIME24 validation sets into `dapo17k.yaml`. The pinned permalink still resolves, but it points at a config that main no longer ships. The other links in this page's tables already use `main`. | | `reference/faq.mdx` (4 dataset upload/migrate examples) | `resources_servers/multineedle/configs/multineedle.yaml` | `resources_servers/example_multi_step/configs/example_multi_step.yaml` | NVIDIA-NeMo#291 renamed `multineedle` to `example_multi_step`. The upload command reads `domain` from this config, and `example_multi_step.yaml` sets it. | Versioned snapshots under `fern/versions/v*/` are frozen and left as they shipped. ### Test `tests/unit_tests/test_fern_docs_links.py::test_cited_resources_server_configs_exist` scans every latest page for `resources_servers/<name>/configs/<file>.yaml` and asserts that each file exists. It skips `my_weather_tool`, which `environment-tutorials/single-step-environment.mdx` has the reader create. It also asserts that it found at least one citation, so a broken regex can't make it pass vacuously. ## Before the fix Gym `974d1a921` (main). The new test fails (`1 failed`) and reports these six citations: ``` fern/versions/latest/pages/contribute/rl-framework-integration/gym-rl-framework-integration-success-criteria.mdx:38: resources_servers/math_with_judge/configs/bytedtsinghua_dapo17k.yaml fern/versions/latest/pages/reference/cli-commands.mdx:695: resources_servers/math/configs/math.yaml fern/versions/latest/pages/reference/faq.mdx:41: resources_servers/multineedle/configs/multineedle.yaml fern/versions/latest/pages/reference/faq.mdx:53: resources_servers/multineedle/configs/multineedle.yaml fern/versions/latest/pages/reference/faq.mdx:91: resources_servers/multineedle/configs/multineedle.yaml fern/versions/latest/pages/reference/faq.mdx:108: resources_servers/multineedle/configs/multineedle.yaml ``` The documented `gym env prefetch` example, run with venv installs stubbed out so that only config resolution and server discovery run (`policy_model_name` is normally supplied by `env.yaml`): ``` $ gym env prefetch "+config_paths=[responses_api_models/local_vllm_model/configs/local_vllm_model.yaml,resources_servers/math/configs/math.yaml]" ++policy_model_name=Qwen/Qwen3-4B-Instruct-2507 Error: config_paths entry 'resources_servers/math/configs/math.yaml' was not found. Looked in: - <Gym>/resources_servers/math/configs/math.yaml Check the path is spelled correctly and is relative to your working directory, an extra root (NEMO_GYM_EXTRA_ROOTS / --search-dir), or the Gym install root. exit=1 ``` ## After the fix ``` $ gym env prefetch "+config_paths=[responses_api_models/local_vllm_model/configs/local_vllm_model.yaml,resources_servers/math_with_judge/configs/math_with_judge.yaml]" ++policy_model_name=Qwen/Qwen3-4B-Instruct-2507 Pre-warming venv for: responses_api_models/local_vllm_model Pre-warming venv for: resources_servers/math_with_judge Pre-warming venv for: responses_api_agents/simple_agent Pre-warming venv for: environment_servers/legacy_agent Prefetch complete. exit=0 ``` `gym env resolve --config` now finds `dapo17k.yaml` and `example_multi_step.yaml`. It then stops at the usual unset-required-values check (`policy_model` / judge name), the same as any config resolved without an `env.yaml`. ## Validation - `pytest tests/unit_tests/test_fern_docs_links.py tests/unit_tests/test_opensandbox_documented_config.py`: 25 passed, 21 subtests passed. - `pytest tests/unit_tests/test_cli_main.py`: passed (the `env.py` change only touches a docstring). - `pre-commit run --all-files`: all hooks pass. - Rollouts: N/A. The only change outside docs and tests is the example text in one docstring. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: qiaochuz <qiaochuz@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds hf pointers and descriptions for the new datasets with super release into the configs + docs.