[1/4] feat(environments): define manifest contract by Glorf · Pull Request #2417 · NVIDIA-NeMo/Gym

Glorf · 2026-08-07T18:09:46Z

Summary

Defines an opt-in manifest.yaml contract for Gym environments and benchmarks without changing how workloads run.

Stack

1/4 — Manifest contract (this PR)
2/4 — Static manifest validation
3/4 — Profile-aware scaffolding
4/4 — Onboarding catalog and commands

Each PR contains one commit and is based on the preceding branch.

What this PR delivers

A strict manifest model for identity, catalog metadata, reward semantics, determinism, integration shape, and config-owned composition mirrors.
Four integration profiles matching Gym's existing extension points: custom-gym-verifier, custom-gym-agent-loop, external-agent-loop, and external-rollout-driver.
A Pydantic contract that can emit Draft 2020-12 JSON Schema on demand.
SPDX expression validation through packaging, including LicenseRef values and Gym's private-license labels.
A neutral example, focused contract tests, and contributor documentation.

The manifest is the single place to read a workload declaration. Gym configuration remains the authoritative place to write runtime composition. Names and versions must be non-empty; Semantic Versioning is documented guidance, while immutable-version enforcement is staged work.

Compatibility and scope

Existing environments and benchmarks continue to run without a manifest.
No runtime dispatch, wire protocol, server route, or existing workload changes.
The manifest remains opt-in until later stack layers consume it.
Capabilities, concrete version resolution, pinning, swap enforcement, and certificate-backed validation remain follow-ups.

Validation

Manifest parsing, invariants, SPDX expressions, serialization, and schema tests.
Independent Draft 2020-12 validation of the generated representation.
Ruff and whitespace checks.

copy-pr-bot · 2026-08-07T18:09:49Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-08-07T18:10:55Z

🌿 Preview your docs: https://nvidia-preview-mbien-environment-onboarding-foundation.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

nemo/gym/contribute/environments/new-environment

ffrujeri · 2026-08-07T21:26:41Z

profile-aware scaffolding for stock-loop, measured-loop, external-loop, and custom-driver;

Those names are quite confusing to me, not sure I understand their meaning without looking into details

Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary Adds service-free validation for manifest-backed environments and benchmarks. It checks the declaration against resolved Gym configuration before any service starts. ## Stack - 1/4 — [Manifest contract](#2417) - **2/4 — Static manifest validation** (this PR) - 3/4 — [Profile-aware scaffolding](#2442) - 4/4 — [Onboarding catalog and commands](#2443) This PR contains one commit and is based on the 1/4 branch. ## What this PR delivers - Offline resolution of Gym configuration and inheritance without endpoint probes, service startup, or model calls. - Manifest identity and catalog-path checks. - Config-authoritative component and dataset selection, including an actionable error when composition has more than one dataset-bearing agent. - Read-only comparison of manifest composition mirrors with resolved config, plus explicit atomic synchronization. - Advisory integration-profile inference with evidence, warnings for mismatches, and `unknown` when static source inspection is inconclusive. - Static parsing of preparation and rollout-driver hooks, plus streamed JSONL and prompt-contract validation. ## Compatibility and scope - Gym config remains the runtime source of truth; manifest mirrors never select runtime composition. - Validation does not import workload components, execute preparation code, start services, provision compute, call a model, probe endpoints, or inspect output-directory writability. - Profile inference reports evidence and never dispatches or gates runtime behavior. - Existing manifest-free workloads continue to run unchanged. ## Validation - Representative built-in and custom profile-inference cases. - Mirror repair, multi-agent ambiguity, hook parsing, prompt application, and malformed-data regressions. - Ruff and whitespace checks. ## Next layer The next PR adds non-destructive profile-aware scaffolding and reusable verifier fixtures. Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary Adds non-destructive scaffolding for all four integration profiles and a reusable, in-process verifier-fixture contract. ## Stack - 1/4 — [Manifest contract](#2417) - 2/4 — [Static manifest validation](#2441) - **3/4 — Profile-aware scaffolding** (this PR) - 4/4 — [Onboarding catalog and commands](#2443) This PR contains one commit and is based on the 2/4 branch. ## What this PR delivers - Runnable starting points for environments and benchmarks across `custom-gym-verifier`, `custom-gym-agent-loop`, `external-agent-loop`, and `external-rollout-driver`. - The appropriate authored extension point for each non-default profile while runtime wiring remains config-driven. - Conservative profile reporting: generated delegation placeholders infer as `unknown` until their authored behavior is implemented. - Benchmark source data, prompt configuration, and a typed `prepare()` hook. - A resources-server-owned verifier fixture covering full reward, zero reward, malformed input, and seeded determinism. - `--reuse-verifier` support across all profiles for compatible existing scorers with an explicit reward contract. - One canonical resource-component renderer shared with the existing `gym env init --resources-server` path; the legacy command keeps its combined runnable config. - Conflict-free reruns, atomic fixture updates, and rejection of unsafe paths, invalid Python names, or unsupported reused compositions. ## Compatibility and scope - Profiles classify authoring boundaries; they do not change runtime dispatch. - Existing workloads do not need manifests or fixtures to keep running. - No existing environment or resources-server implementation is modified. - Identical scaffold reruns are no-ops, and any conflict aborts the complete write set. ## Validation - Both workload kinds across all four profiles. - Generated-code lint and format checks. - Reused scorer contracts, reward direction, malformed input, determinism, path safety, and atomic-update regressions. ## Next layer The final PR exposes the stack through discovery and the four-command onboarding journey. Signed-off-by: Michal Bien <mbien@nvidia.com>

## Summary Completes the user-facing onboarding MVP with a local registry and four commands: ```bash gym env init --environment my_eval --profile custom-gym-verifier gym env validate my_eval gym env test my_eval gym env publish my_eval ``` ## Stack - 1/4 — [Manifest contract](#2417) - 2/4 — [Static manifest validation](#2441) - 3/4 — [Profile-aware scaffolding](#2442) - **4/4 — Onboarding catalog and commands** (this PR) This PR contains one commit and is based on the 3/4 branch. ## What this PR delivers - A unified local workload registry for manifest-backed and legacy environments and benchmarks. - Transition visibility for runnable compositions still colocated under `resources_servers/`; reusable scorer, tool, and state components without agent datasets remain component entries. - Kind-filtered list, search, inspect, status, profile, domain, and JSON views. - Environment and benchmark targets for `gym env init`. - Static checks, advisory profile evidence, warnings, and mirror synchronization through `gym env validate`. - Verifier execution in the resources server's managed dependency environment through `gym env test`. - A local structural `gym env publish` check that reruns validation and fixtures, rejects manifest metadata placeholders, and confirms exact catalog discovery as `experimental`. - Public documentation for terminology, ownership, profiles, migration behavior, and all four commands. `publish` does not assess implementation quality, run a full evaluation, deploy, commit, push, or claim certificate-backed validation. ## Compatibility - Existing runtime configuration and execution remain unchanged. - Existing init, config validation, resources-server test, list, and search forms remain available. - Manifest-free runnable workloads remain visible with `no-manifest` status and continue to run as before. - Malformed manifests are isolated during global discovery and remain directly diagnosable with `gym env validate --manifest ...`. ## Validation - 510 sandbox-safe onboarding and affected CLI tests pass; the socket-binding case also passes outside the sandbox. - CI-version Ruff and whitespace checks pass. ## Staged follow-ups - Capability declarations, concrete component-version resolution, pinning, and swap refusal. - Runtime readiness probes as a separate pre-run validation layer. - In-place manifest backfill and phased CI enforcement for legacy runnable compositions. - Repository publication automation, ownership updates, and immutable version enforcement. - Certificate-backed `validated` status and a hosted catalog index. - A session-stateful onboarding sentinel and stored-trajectory replay coverage. These follow-ups strengthen runtime readiness, governance, and migration. They do not change the manifest contract, guided authoring path, local checks, or transition registry delivered here. Signed-off-by: Michal Bien <mbien@nvidia.com>

Glorf marked this pull request as ready for review August 7, 2026 18:10

Glorf requested a review from a team as a code owner August 7, 2026 18:10

Glorf force-pushed the mbien/environment-onboarding-foundation branch from 294861c to fad2d1f Compare August 7, 2026 18:10

Glorf force-pushed the mbien/environment-onboarding-foundation branch from fad2d1f to a6ab75d Compare August 7, 2026 20:04

Glorf changed the title ~~feat(environments): add manifest-backed onboarding workflow~~ feat(environments): add manifest-backed onboarding foundation Aug 7, 2026

ritaneves requested a review from bxyu-nvidia August 8, 2026 08:23

Glorf force-pushed the mbien/environment-onboarding-foundation branch from a6ab75d to 91ff25d Compare August 10, 2026 08:47

Glorf changed the title ~~feat(environments): add manifest-backed onboarding foundation~~ feat(environments): define manifest contract Aug 10, 2026

github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Aug 10, 2026

github-actions Bot mentioned this pull request Aug 10, 2026

[Tracker] Pull request handoff SLA #2067

Open

Glorf force-pushed the mbien/environment-onboarding-foundation branch from 91ff25d to 607f665 Compare August 10, 2026 11:41

Glorf changed the title ~~feat(environments): define manifest contract~~ [1/4] feat(environments): define manifest contract Aug 10, 2026

This was referenced Aug 10, 2026

[2/4] feat(environments): add static manifest validation #2441

Merged

[3/4] feat(environments): add profile-aware scaffolding #2442

Merged

[4/4] feat(environments): add onboarding catalog and commands #2443

Merged

bxyu-nvidia requested changes Aug 11, 2026

View reviewed changes

Comment thread examples/environment_manifest.yaml Resolved

Comment thread nemo_gym/environment/schemas/environment-manifest.schema.json Resolved

Comment thread nemo_gym/environment/manifest.py Resolved

Comment thread nemo_gym/environment/manifest.py Resolved

github-actions Bot removed the sla:review-overdue Review response is over the one-business-day SLA label Aug 11, 2026

Glorf force-pushed the mbien/environment-onboarding-foundation branch 3 times, most recently from d359fdd to c191f82 Compare August 11, 2026 12:51

Glorf requested a review from bxyu-nvidia August 11, 2026 12:53

bxyu-nvidia approved these changes Aug 11, 2026

View reviewed changes

ritaneves force-pushed the mbien/environment-onboarding-foundation branch from c191f82 to 14f6033 Compare August 11, 2026 15:58

kajalj22 approved these changes Aug 11, 2026

View reviewed changes

feat(environments): define manifest contract

fc34da5

Signed-off-by: Michal Bien <mbien@nvidia.com>

Glorf force-pushed the mbien/environment-onboarding-foundation branch from 14f6033 to fc34da5 Compare August 11, 2026 18:29

Glorf enabled auto-merge (squash) August 11, 2026 18:30

Glorf merged commit 1c56cba into main Aug 11, 2026
23 of 26 checks passed

Glorf deleted the mbien/environment-onboarding-foundation branch August 11, 2026 18:32

Glorf mentioned this pull request Aug 25, 2026

feat: init environment CLI with composable scaffolding #1029

Closed

1 task

Navigation Menu

Repository navigation

[1/4] feat(environments): define manifest contract - #2417

[1/4] feat(environments): define manifest contract#2417
Glorf merged 1 commit into
mainfrom
mbien/environment-onboarding-foundation

Glorf commented Aug 7, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Aug 7, 2026

Uh oh!

github-actions Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

ffrujeri commented Aug 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Glorf commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stack

What this PR delivers

Compatibility and scope

Validation

Uh oh!

copy-pr-bot Bot commented Aug 7, 2026

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ffrujeri commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Glorf commented Aug 7, 2026 •

edited

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading

ffrujeri commented Aug 7, 2026 •

edited

Loading