[prokotg]

Summary

• A guide for contributors adding an exporter for gym eval export: contribute/exporters.mdx, with a card on the Contribute overview.
• It covers how an export runs, the Exporter contract (RunArtifacts, the settings model, ExportResult states and exit codes), registration in run_exporter_registry.py and what gym eval submit adds, when to choose an in-process or an isolated exporter, testing, and credential handling.
• Two worked examples: an in-process summary exporter (named per run, published atomically, every filesystem error returned as failed), and an isolated acme exporter with its worker file. atif and mlflow are pointed to as real references.
• The example code lives in tests/unit_tests/exporter_guide/. Tests run it (the isolated one against a fake acme_sdk through the real worker), and a test checks that the page's code blocks match those files exactly.

Docs only; no behavior change. Part of #2977 (export-aware evaluation runs).

PR plan

Separate PRs; steps 1-8 are stacked (stack #4910). Goal: make gym eval export pluggable so a finished run can go to tracking systems without live exporters, and gym eval submit can export after each benchmark.

1. save the masked, unresolved config from gym eval run (feat(eval): save the masked config beside rollouts from gym eval run #4908)
2. RunArtifacts and path derivation helpers (feat(eval): add RunArtifacts to derive a run's files from its rollouts path #4909)
3. Exporter base class and registry, starting with atif, plus the positional exporter CLI argument (--format deprecated) (feat(eval): add the Exporter base class and registry, with atif as the first exporter #4911)
4. post-run mlflow and wandb export: shared metrics flattening, BaseExporter.export() and the already-exported guard (feat(eval): export a finished run to MLflow or W&B with gym eval export #4914)
5. isolated-venv subprocess runner for directory exporters (feat(eval): add IsolatedExporter to run an exporter's backend work in its own venv #4915)
6. a second isolated-venv exporter (feat(eval): export a finished run to EvalRDL with gym eval export evalrdl #4965)
7. gym eval submit integration: an export map with job-level defaults and per-benchmark overrides, validation, install extras, and a job-script tail (Slurm only at first) (feat(orchestration): run exports after each benchmark of gym eval submit #4974)
8. This PR: contributor guide: add an exporter (user-facing docs land in each step's PR)

Validation

• test_exporter_guide.py: 18 passed (the examples also cover run-specific output names, atomic publishing and filesystem errors returned as failed). pre-commit: passed.
• Changing an example file makes the page-sync test fail. Following the page's "Register It" steps with the real CLI, gym eval export summary ... succeeded and a second run reported skipped (the temporary registry entries were reverted).

Rollout evidence

N/A: documentation and tests only.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[prokotg]

/ok to test 5e743f2

[prokotg]

/review

[svcnvidia-nemo-ci]

SHIP

LGTM — no remaining eligible findings in this follow-up.

Verified all four previously reported failures are resolved in current source:

• The summary exporter distinguishes the reported same-filename runs, publishes through a temporary sibling, and returns expected input/output filesystem failures as FAILED.
• Acme preparation now guards precondition checks, directory creation, and copying before environment construction.

Checked the regression and preservation assertions, mirrored guide blocks, and directly affected exporter/worker contracts. The worker-environment stub accepts the interpreter argument used by production. The summary-I/O topic was independently checked because its delegated report was unavailable.

This was a targeted follow-up, not a fresh whole-PR review. The previously reviewed tree was unavailable, limiting inter-review comparison to supplied historical evidence and current source. No eligible new regression was established. Static inspection only; tests and the Fern build were not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

[prokotg]

/review

[svcnvidia-nemo-ci]

SHIP

LGTM — no remaining eligible findings in this follow-up.

Verified the four previously reported failures remain resolved in current source:

• The summary exporter distinguishes the reported same-filename runs, publishes only complete output, and returns input/output filesystem failures as FAILED.
• Acme preparation guards precondition checks, directory creation, and copying before environment construction.

Checked the relevant regression and preservation assertions, mirrored guide examples, and directly affected contracts for submit settings, credentials, worker launch/timeouts, and completion markers. The summary and Acme/worker topics were checked directly because two delegated reports were unavailable.

This was a targeted follow-up, not a fresh whole-PR review. The previously reviewed tree was unavailable, limiting inter-review comparison to supplied historical evidence and current source. No eligible introduced regression was established. Static inspection only; tests and the Fern build were not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

[prokotg]

/ok to test f8cdcc7

[prokotg]

/review

[svcnvidia-nemo-ci]

SHIP

LGTM — no remaining eligible findings in this follow-up.

Verified that the four previously reported failures remain fixed:

• The summary exporter distinguishes the reported same-filename runs, publishes only complete output, and returns expected input/output filesystem failures as FAILED.
• Acme preparation guards precondition checks, directory creation, and copying before environment construction.

Checked the relevant regression and preservation assertions, mirrored examples, and affected contracts for submit settings, credential handling, worker launch/timeouts, completion markers, and per-export result retention. The summary and Acme/worker topics were checked directly because their delegated reports were unavailable.

This was a targeted follow-up, not a fresh whole-PR review. The previously reviewed tree was unavailable, limiting inter-review comparison to supplied historical evidence and current source. No eligible introduced regression was established. Static inspection only; tests and the Fern build were not executed.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

[prokotg]

/ok to test 4f2d01c

[prokotg]

/ok to test 240efbc

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41672241110