[adil-a]

Stacked on #4865 (M31). Second slice of M38; the first slice is #4827.

Summary

gym dataset pull (M38, second slice): fetch a Harbor dataset into environments/harbor/ without preparing or running anything, and a name mapping between a hub dataset and its committed entry so one dataset never lives in two sibling folders. Publishing to the hub is deferred to a future work item.

• gym dataset pull harbor:<org>/<name>[@<tag> | @sha256:<digest>] and gym dataset pull harbor:<dataset>[@<version>] run the existing package-store and registry fetches (fetch_ref, fetch_package_dataset, the M9/M33 claim and manifest machinery) and print the folder, the task count and the dataset digest; a stale, edited or foreign folder is refused in the same words as the fetch inside gym eval run; a second pull of an unchanged dataset downloads nothing and says "unchanged since the last pull" (a registry reference whose version the registry picks is resolved before the fetch, so that form says it too). The command adds no flag (the registry cache is refreshed by deleting environments/harbor/.harbor/registry.json, as before).
• A committed entry declares its upstream in its own dataset.toml: the new [gym] source key (a harbor: reference, validated by the hub's own parser). gym dataset pull <entry> (for example terminal_bench_4) fetches it into the entry's own folder, in whichever search root commits it, where the task folders and manifest.toml are gitignored beside the committed README, YAML and dataset.toml. A pull of the declared reference itself, and gym eval run / gym dataset validate on it, land in the same folder; the match is exact (latest and a tag are different references). A reference no entry declares lands in environments/harbor/<hub-folder-name>/ as before.
• gym dataset validate <entry> and gym eval run <entry> ... resolve the bare entry name to that folder without fetching; an entry not pulled yet names the pull to run, task folders there with no fetch record (no manifest.toml, or one of another source) are refused with the pull to run instead of running as a local dataset under the entry's name, and an existing folder of the same name or an explicit path keeps meaning the path for those two commands (the pull takes no path and finds the entry from any working directory).
• NEMO_GYM_DATASETS_DIR still moves every pull: an entry then lands in <datasets dir>/<entry>/ with a copy of the entry's Gym layer beside the tasks (its dataset.toml and whatever dataset-level task_data.py or verifier.py it commits, the files that enter the dataset digest), placed before the first task lands and replaced when the entry's files change (pull, validate and run do it alike, and bring the manifest beside the tasks up to date with the copies at once, so the three never disagree about one folder or its digest); a copy edited in place is refused by all three, never overwritten.
• A pull whose fetch downloaded nothing (every folder in place) still brings the manifest's digest of dataset.toml up to date (manifest.refresh_gym_layer, task records untouched), so the digest it reports after an edit of the entry's table is the one the next run carries and that run rewrites nothing; a change of the Gym layer alone is announced in the manifest check's words (Rewrote manifest.toml: ...; dataset digest <old> -> <new>) whichever step applied it. A fetch or manifest-check refusal that names a committed entry's own folder adds what to delete there (the pulled task folders and manifest.toml; the README, YAML and dataset.toml stay; another version is pulled by changing source); the refusal of an interrupted fetch names gym dataset pull <entry> to run again instead.
• The two committed entries declare their sources: terminal_bench_4 pins harbor:terminal-bench/terminal-bench@4.0.0; terminal_bench_2_1 pins the store's TB 2.1 by content hash (@sha256:7d7bdc1c…), since the store tags it latest only.

Behaviour change for existing checkouts. gym eval run harbor:terminal-bench/terminal-bench@4.0.0 now lands in environments/harbor/terminal_bench_4/ instead of terminal-bench-4.0.0/: the dataset is fetched afresh (no folder is moved), the taskset, the rows' task_source and the results/harbor/<taskset>/ path change from terminal-bench-4.0.0 to terminal_bench_4, and the dataset digest changes because the entry's dataset.toml now enters it. While the old folder is still there, the pull and a run of the reference print a Note: naming it; it can be moved aside or deleted. The README, the Harbor tasks page and the TB4 entry README carry the same note.

What changes

• nemo_gym/tasks/harbor/entries.py (new): committed entries (find_entry, committed_entries, entry_for_ref, entry_target), the pull plan (plan_pull), the entry's destination (its own folder unless NEMO_GYM_DATASETS_DIR is set) and the destination with the entry's Gym layer current and its manifest following it (gym_layer, prepare_destination).
• nemo_gym/tasks/harbor/cli.py: pull_dataset, PulledDataset (with the superseded hub folder note), pull_target (the command), entry_folder with its fetch-record check, with_entry_hint; prepare_target fetches a harbor: target through pull_dataset (so the entry mapping applies to gym eval run and gym dataset validate) and resolves an entry name through entry_folder.
• nemo_gym/tasks/harbor/manifest.py: refresh_gym_layer (the manifest's dataset.toml/dataset-files digests recomputed from disk, task digests and pins kept as recorded, written only when changed, announced with the manifest check's Rewrote notice); gym_layer_change and report_rewrite shared with check_manifest.
• nemo_gym/tasks/harbor/hub.py, package_store.py: fetch_ref, fetch_dataset and fetch_package_dataset take an optional destination folder; expected_folder names a reference's folder without resolving it, resolved_folder resolves a versionless registry reference against the cache (for the unchanged report and the note).
• nemo_gym/tasks/harbor/dataset_config.py: GymDatasetConfig.source (validated as a hub reference; provenance only, shapes no task, stays out of the datasets: entry).
• nemo_gym/cli/main.py: gym dataset pull command (TARGET only); TARGET help of dataset validate and eval run names the entry form.
• .gitignore: environments/harbor/*/dataset.toml is committed with the entry's README and YAML (in every folder there, a pulled hub folder included, as README.md and *.yaml already are: a dataset.toml written into an old hub folder by hand now shows as untracked; the docs say to move its keys into the entry's file).
• environments/harbor/terminal_bench_4/dataset.toml, terminal_bench_2_1/dataset.toml (new) and their READMEs; scaffold.py documents the key as a comment.
• Docs: harbor environment server README (fetch paragraph, new "Committed dataset entries" section with the migration note), reference/cli-commands.mdx (gym dataset pull; the gym eval run TARGET sentence names the entry form), data/harbor-tasks.mdx ([gym] source, "gym dataset pull and committed entries", TARGET rows, migration note), TB 2.1 tutorial.

Test plan

• tests/unit_tests/test_harbor_entries.py (new, 56 tests): the source key (accepted forms, refusals naming the key, defaults, the two committed entries' values); entry lookup (with and without source, unreadable table, earlier root wins, an entry in a second search root pulled into its own folder with the env var unset, hub folders skipped); entry_for_ref exact matching and the two-entries refusal; entry_target (bare name, paths and existing folders win); plan_pull; prepare_destination under a moved datasets dir (copy placed, refreshed when the entry's file changes and the copy is the last pull's, an in-place edit refused); the pull through main() against the package store faked at its HTTP boundary (entry pull, report lines, second pull unchanged with no download, reference and entry name are one folder, reference without an entry lands in the hub folder, registry reference through a local git repo, the refresh kwarg reaching the loader and no --refresh-registry flag, moved datasets dir, entry table edited after the pull reaching pull and prepare alike, a copy edited in place refused by pull, validate and run in the same words, an earlier hub folder named and left alone, edited folder refused by pull and by gym eval run alike, foreign manifest refused, one-line parser errors); target resolution in prepare, gym dataset validate <entry> and gym eval run <entry> (pulled, unpulled, sourceless, same-name folder wins, moved datasets dir, edited pull refused at prepare); the record following the copied table under a moved datasets dir through two successive entry edits (registry), around a failed fetch (first fetch failing with no manifest, a fetch failing after a pull) and after a run refused for the agent; a registry entry pulled in place reporting the edited table's digest with no "Rewrote" at the next prepare; refusals naming a committed entry's own folder ending with what to delete there (pull by name and by reference, run by reference, run by name after a hand edit; no hint under a moved datasets dir); a folder of the entry's name in the working directory not hiding the entry from the pull. tests/unit_tests/test_harbor_manifest.py: refresh_gym_layer (follows dataset.toml, keeps task digests and pins including a pin without a digest, leaves a pins-only manifest and a missing one alone).
• Two existing fakes updated for the new folder= parameter (test_harbor_tasks.py, test_harbor_package_store.py).
• Repair round 3 adds: test_a_pull_that_lost_its_manifest_is_refused_not_run_as_local (in-place and under a moved dir), the interrupted-fetch branch of test_a_refusal_naming_the_entry_folder_says_what_to_delete_there, test_resolved_folder_... and test_a_registry_reference_without_a_version_reports_unchanged_on_the_second_pull, test_a_moved_datasets_dir_gets_the_entry_s_dataset_level_files_too, test_a_pull_whose_fetch_rewrote_the_gym_layer_announces_the_digest_change, the exact-notice assertions in test_refresh_gym_layer_... and test_a_registry_entry_pulled_in_place_..., and test_global_config.py::test_cli_parse_restores_hydras_args_parser_instead_of_nesting_wrappers (the one change outside nemo_gym/tasks/harbor/).
• Env server suite (environment_servers/harbor/tests -k "not partial_eval and not buildah and not validate_ and not real_image"): 470 passed, 2 deselected. Oracle suite: 38 passed. The ten touched core unit files plus test_global_config.py, in one session: see REPORT.md for the count.
• ruff check, ruff format --check, pre-commit run --files <changed files>: clean.
• Real, with NEMO_GYM_DATASETS_DIR on Lustre (the home quota on the build node is full): gym dataset pull terminal_bench_4 end to end through the entry (66 tasks, digest 42f85927f0ad…; second pull and the bare harbor:terminal-bench/terminal-bench@4.0.0 pull both land in the entry's folder and report unchanged); the pinned TB 2.1 reference pulled into the terminal_bench_2_1 entry and three of its tasks validated with the oracle on OpenSandbox through the entry name (all 1.0); the plain harbor:terminal-bench/terminal-bench-2-1 reference pulled into terminal-bench-2-1/ and pulled again unchanged; the entry's dataset.toml edited after the pull → the next pull refreshes the copy and the digest follows (no download), restored → digest back; the copy edited in place → gym dataset pull and gym dataset validate refuse in the same one line with nothing written; an old terminal-bench-4.0.0/ beside the entry → the Note: line, gone once moved aside. Repair round 2, same layout: four successive edits of the entry's dataset.toml with a pull between each (one with the package store unreachable), every pull's manifest dataset_toml_digest equal to the copy's digest and the second, third and fourth edits accepted (the case that was refused before); after the fifth edit and its pull, gym dataset validate terminal_bench_2_1 --only-tasks fix-git --sandbox opensandbox through the entry name printed no Rewrote line and the oracle solved the task (1.0, exit 0); gym dataset pull terminal_bench_4 from benchmarks/ (which holds terminal_bench_4/) pulls the entry. Repair round 3, datasets dir node-local (/raid/scratch): the TB 2.1 pin pulled fresh (89 tasks), its manifest.toml moved aside, gym dataset validate/eval run by entry name both refused with the pull named and wrote nothing, the pull then recorded the kept folders in 2 s (no re-download) and a second pull said "unchanged"; an edit of the entry's dataset.toml made the pull print exactly one Rewrote ...; dataset digest <old> -> <new> and the following unchanged pull print none. See REPORT.md.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally.
• Documentation added or updated.
• Pre-commit checks pass locally on the changed files.
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[adil-a]

/ok to test 2f53402

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4872-fb511ae6d921.docs.buildwithfern.com/nemo/gym

[adil-a]

/ok to test d843dd8

[adil-a]

/ok to test a9a4ffc

[adil-a]

/ok to test 1789ce5

[adil-a]

/ok to test ac4a014

[adil-a]

/ok to test a100ed1

[yuhezhang-ai]

Thanks, Adil. I rechecked the latest revision against #4865. The five inline findings are addressed, including protecting copied configuration after preparation without NEMO_GYM_DATASETS_DIR. The reference-spelling and moving registry-pin fixes also look good, and the focused regression tests passed.

The remaining cleanup assertion still checks terminal-bench-4.0.0/alpha rather than terminal_bench_4/alpha. The fault-injection reproduction still passes with a partial file in the actual destination. Please correct that assertion or explicitly isolate the unmapped-reference case; I consider this non-blocking.

The PR description also still references REPORT.md and a test_global_config.py change absent from this diff. Please clarify those validation references.

[yuhezhang-ai]

Correction to my earlier follow-up: I missed the installed-package destination follow-up explicitly deferred from #4827 to this PR. The five resolved findings remain fixed, but the combined stack should wait for the writable entry-destination fallback and its installed-layout regression. Keeping that fix in #4872 is fine. The cleanup-test and validation-description comments remain non-blocking.

[adil-a]

/ok to test 05e31e4

[adil-a]

/ok to test 12a629c

[ffrujeri]

LGTM, thanks.

[adil-a]

Re the correction above: the installed-package destination fallback deferred from #4827 is now in this PR (head 12a629c). DatasetEntry.destination falls back to datasets_dir()/<entry> for an entry committed outside the working root (site-packages for a package install, or a NEMO_GYM_EXTRA_ROOTS root), the pull copies the entry's Gym layer there, and test_installed_entries_are_pulled_under_the_working_directory_not_the_installation rebuilds the installed layout and covers both target forms with the variable unset, asserting the installation directory is untouched. Details in the resolved thread on entries.py.

[adil-a]

/ok to test f43ce06

[adil-a]

/ok to test 849495a

[adil-a]

/ok to test 3fafede

[ffrujeri]

LGTM.

[adil-a]

/ok to test 9062904

[adil-a]

/ok to test 816e65b

[yuhezhang-ai]

Thanks, Adil. Rechecked 816e65bc against the current #4865 parent. Both original installed-package reproductions now pass unchanged: downloads leave a read-only project-local .venv untouched, and later installed-entry changes reach the existing download.

The focused regressions also confirm that edited copies stay protected and user-authored entries retain precedence. I have no remaining blocking findings; the installed-package thread can stay resolved.

The cleanup-test assertion and outdated validation references remain non-blocking follow-ups.

[adil-a]

/ok to test 4006449

[adil-a]

/ok to test bc68ab5

[ffrujeri]

LGTM.

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40540366340

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40546615675

[adil-a]

/ok to test 8034d45

[yuhezhang-ai]

Rechecked 8034d456 against the current #4865 parent, 90a15c96. This PR's patch is unchanged from the previously approved 816e65bc; the installed-package download and refresh fixes, copy protection, and earlier resolved findings remain intact.

Reused the completed focused regression validation after confirming the entry/download implementation and its tests are unchanged. Current core/server CI, lint, docs and wheel validation passed. I have no remaining blocking findings. The cleanup-test assertion and outdated validation references remain non-blocking follow-ups.

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40612810697

[adil-a]

/ok to test 60b54e6

[adil-a]

/ok to test fb511ae

[yuhezhang-ai]

Rechecked fb511ae6 against the current #4865 parent, 6c0fbfef. The PR-specific patch is unchanged from the previously approved revision, and the earlier fixes remain intact. No remaining blocking findings.

The cleanup-test assertion and outdated validation references remain non-blocking follow-ups.

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40725069900