feat(eval): export a finished run to MLflow or W&B with gym eval export by prokotg · Pull Request #4914 · NVIDIA-NeMo/Gym

prokotg · 2026-10-22T12:33:26Z

Summary

gym eval export mlflow|wandb sends a finished run to the backend: its saved config, aggregate metrics and (optionally) rollouts. BaseExporter becomes an Exporter with export(artifacts), reusing the live exporters' hooks.
Guard: if <stem>_<exporter>_export_done exists the run is skipped without contacting the backend. A post-run export writes it only after the whole run is sent (atomically, holding the backend run id), so a failed export can be retried and a failed write leaves none. It is separate from the <stem>_<exporter>_run_id a live run records for resume. A live run id does not make the export skip, because it only shows that the run opened, not that every upload landed: the export goes ahead as a separate run, logs a warning and returns live_run_id. gym eval submit already rejects a live exporter plus the same export for one backend, so only a manual export after a live run is affected. A post-run export never reads or writes the live run id, so a later live resume does not reopen the export's run.
It fails before opening the backend unless _aggregate_metrics.json exists, is not older than the rollouts and records no failed agent (a partial result is never exported), and the saved config exists.
Rollouts upload unless the saved upload_rollouts is false (only that value is resolved, so an interpolated setting works); upload_rollouts= on the command overrides. Settings and credentials come from the export command, never from the masked saved config.
flatten_aggregate_metrics replaces two duplicate blocks in collection and reverification.
A WANDB_API_KEY in the environment is used when the command sets no key.
skip_exporters parser option, so the export command does not also open a live exporter; a missing SDK error names the extra to install. CLI reference and the live exporter settings page updated.

Part of #2977 (export-aware evaluation runs).

PR plan

Separate PRs; steps 1-8 are stacked (stack #4910). Goal: make gym eval export pluggable so a finished run can go to tracking systems without live exporters, and gym eval submit can export after each benchmark.

save the masked, unresolved config from gym eval run (feat(eval): save the masked config beside rollouts from gym eval run #4908)
RunArtifacts and path derivation helpers (feat(eval): add RunArtifacts to derive a run's files from its rollouts path #4909)
Exporter base class and registry, starting with atif, plus the positional exporter CLI argument (--format deprecated) (feat(eval): add the Exporter base class and registry, with atif as the first exporter #4911)
This PR: post-run mlflow and wandb export: shared metrics flattening, BaseExporter.export() and the already-exported guard
isolated-venv subprocess runner for directory exporters (feat(eval): add IsolatedExporter to run an exporter's backend work in its own venv #4915)
EvalRDL exporter (feat(eval): export a finished run to EvalRDL with gym eval export evalrdl #4965)
gym eval submit integration: an export map with job-level defaults and per-benchmark overrides, validation, install extras, and a job-script tail (Slurm only at first) (feat(orchestration): run exports after each benchmark of gym eval submit #4974)
contributor guide: add an exporter (user-facing docs land in each step's PR) (docs(contribute): add a guide to writing an exporter for gym eval export #5077)

Validation

42 tests in test_record_exporters.py: export flow, done-marker timing and atomic write, its own file apart from the live run id, a live run id reported and not skipping, skip, failure and retry, preconditions (including a failed agent in the aggregate file), upload intent (including interpolated settings), W&B key from the environment, skip_exporters, missing-SDK error, mocked MLflow and W&B exports. pre-commit: passed.
Real MLflow run (file store, MLFLOW_ALLOW_FILE_STORE=true): one FINISHED run with the flattened metrics, params from the saved config, and global_config.json and rollouts.json artifacts; a second invocation printed skipped. The same real local store also covered: a live run then an export (two runs, the live marker untouched), a live resume (reopens the live run, not the export's), a fresh live run, and a failed upload then a retry (no marker, the retry succeeds). W&B is mocks only.

Known gaps

A run logged live and then exported by hand ends up in the backend twice (see the guard). Skipping safely would need the live path to record its own completion at every export site.
Live runs also log coverage/* metrics; they are not in the aggregate file, so post-run export omits them.
A backend failure partway through leaves a partial MLflow run marked FINISHED; the retry creates a new one.

Rollout evidence

N/A: no agent, verifier or model behavior is touched. Live logging behaves as before; only the metrics flattening moved into a shared function.

copy-pr-bot · 2026-10-22T12:33:30Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

prokotg · 2026-10-22T12:38:08Z

/ok to test 5f0d7c1

prokotg · 2026-10-22T12:38:13Z

/review

github-actions · 2026-10-22T12:40:26Z

🌿 Preview your docs: https://nvidia-preview-pr-4914-0e4dcd944e7a.docs.buildwithfern.com/nemo/gym

svcnvidia-nemo-ci

HOLD — four RISK findings affect export completeness, configuration handling, retryability, and W&B authentication. One NOTE corrects the credential-source documentation.

Reviewed all 10 changed files and traced the affected CLI, artifact-writing, backend, and regression-test paths. This was a static review; tests and backend calls were not executed. Suggested patches and proposed checks require verification before adoption.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

prokotg · 2026-10-23T09:53:43Z

/ok to test 88ecef4

prokotg · 2026-10-23T13:17:01Z

/ok to test d9d5c32

AdamRajfer

Two targeted comments on export completion state and integration with #5029. The second concerns inherited live behavior.

AdamRajfer · 2026-10-27T12:58:28Z

+        marker = artifacts.exporter_run_id_fpath(self.name)
+        if marker.exists():
+            return ExportResult.skipped(reason="already exported", run_id=marker.read_text().strip())


This marker proves that a tracking run was opened, not that its uploads completed. If live rollout upload fails, evaluation can still finish with valid artifacts, but a later gym eval export mlflow returns 'already exported' without attempting the missing upload. I reproduced that with a fake backend; removing the marker lets export succeed. Could we distinguish resumable run identity from completion evidence for the requested export? The live ID still needs to be written during setup for resume.

Good catch, thanks for reproducing it. A post-run export now has its own marker, <stem>_<name>_export_done (written atomically, only after everything was sent) and doesn't touch <stem>_<name>_run_id anymore, so the live path keeps the run id file to itself. Fixed in bbcf507.

One behavior change: a live run id no longer makes gym eval export skip. The export goes ahead as a separate run, logs a warning and returns live_run_id. So a run logged live and then exported by hand can show up twice, which I think beats a silent skip, since the live id only says the run opened. gym eval submit already refuses live + the same export for a backend, so it can't hit this. Making the live path record real completion would mean touching every export site in collection and reverification, so I left that out of this PR.

Covered by tests, and I ran live-then-export, export-then-live-resume, re-export and failed-then-retried upload against a real local MLflow store.

AdamRajfer · 2026-10-27T12:58:28Z

+        if self._exported_artifacts is not None:
+            return None


On the live path this still reopens an existing ID when resume_from_cache=False, so a fresh evaluation at the same output path logs into the previous evaluation. #5029 fixes that decision, including resubmit and cache-miss fallback. I suggest merging #5029 first and rebasing this stack on main. Here, retain the post-hoc return None first, then apply #5029's continuation check to live calls; keep pending post-hoc ID publication separate.

Done, the stack is rebased and merged with it the way you described: post-run return None first, your continuation check for live calls, and the pending post-run id kept separate. On top of that the post-run export no longer shares the run id file with the live path (see the other thread).

prokotg · 2026-10-27T15:43:20Z

Blocked by #5029 (live exporters: reopen the tracking run only on resume, log each landed rollout once).

Both PRs rewrite _recorded_run_id and share the <stem>_<name>_run_id marker, and #5029 changes when a live run reopens it. Landing it first lets this PR rebase once against the final run-id handling. When it does, the early None for post-run exports must stay ahead of #5029's flag checks, so an export never reopens a run.

Removing blocked once #5029 is in.

prokotg · 2026-10-28T09:01:37Z

/review

BaseExporter gains export(artifacts): it logs the run's saved config, aggregate metrics and rollouts through the same hooks live runs use, skips a run it already exported, and writes the run id marker only after the whole run is sent. Live and post-run metrics share one flattener. Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>

…inished runs Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>

…mplete Refuse an aggregate file with a failed agent instead of exporting a partial result, resolve only the upload setting of an interpolated saved config, publish the already-exported marker atomically, keep a WANDB_API_KEY from the environment when the command sets none, and name env.yaml in the docs. Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>

…ide and the upload filter Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>

…y users Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>

…re not uploaded The rollouts file is now only required when the rollouts are uploaded, and the freshness check compares against it only when it is there. The error for a missing file says how to skip the upload. Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>

…tead of writing an empty marker Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>

…'s run id A live run writes its run id when the backend opens, so that file shows the run opened, not that every upload landed. Sharing it as the already-exported marker made gym eval export skip a run whose live upload had failed, and made a later live resume reopen the export's run. A post-run export now writes <stem>_<name>_export_done and ignores the live run id, logging and reporting it when there is one. The live path reads and writes only <stem>_<name>_run_id. Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>

prokotg · 2026-10-28T09:14:41Z

/ok to test 0e4dcd9

prokotg · 2026-10-28T09:14:51Z

/review

svcnvidia-nemo-ci

SHIP — no eligible findings remain in this follow-up.

Verified against current source and regression assertions:

Failed-agent aggregates are rejected before backend setup; repaired aggregates export both agents. Selective upload-setting resolution preserves canonical-key precedence and unrelated unresolved expressions (implementation, regressions).
Completion-marker publication remains atomic, with partial-write failure leaving no marker and allowing retry. Post-run completion markers are separate from live resumable IDs (publication, regressions).
W&B retains an inherited API key unless command configuration overrides it (setup, credential assertions). The documentation names env.yaml, matching the parser.

Also checked optional-rollout export, missing backend IDs, command/live setup separation, and live-resume isolation along their directly affected paths. No eligible introduced regression was established.

This was a static follow-up of review 5663786908, not a fresh whole-PR review. The previously reviewed tree was unavailable, limiting exact inter-review comparison. One delegated report was unavailable; its artifact-readiness/upload topic was independently inspected by the parent. Tests and backend calls were not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

github-actions · 2026-10-28T12:08:05Z

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40685324258

prokotg added this pull request to stack #4910 October 22, 2026 12:33

svcnvidia-nemo-ci added area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters feature New capabilities, enhancements, or enablement work labels Oct 22, 2026

copy-pr-bot Bot temporarily deployed to public October 22, 2026 12:38 Inactive

copy-pr-bot Bot temporarily deployed to public October 22, 2026 12:40 Inactive

copy-pr-bot Bot temporarily deployed to public October 22, 2026 12:41 Inactive

prokotg mentioned this pull request Oct 22, 2026

feat(eval): add IsolatedExporter to run an exporter's backend work in its own venv #4915

Merged

svcnvidia-nemo-ci reviewed Oct 22, 2026

View reviewed changes

Comment thread nemo_gym/exporters/base.py Resolved

Comment thread nemo_gym/exporters/base.py Resolved

Comment thread nemo_gym/exporters/base.py Resolved

Comment thread nemo_gym/exporters/base.py Resolved

Comment thread fern/versions/latest/pages/reference/cli-commands.mdx Resolved

prokotg force-pushed the prokotg/eval-export-record-exporters branch from 5f0d7c1 to 88ecef4 Compare October 23, 2026 09:52

copy-pr-bot Bot temporarily deployed to public October 23, 2026 09:54 Inactive

copy-pr-bot Bot temporarily deployed to public October 23, 2026 09:56 Inactive

copy-pr-bot Bot temporarily deployed to public October 23, 2026 09:57 Inactive

prokotg mentioned this pull request Oct 23, 2026

feat(eval): export a finished run to EvalRDL with gym eval export evalrdl #4965

Merged

v0ropaev mentioned this pull request Oct 23, 2026

fix(cli): quote the eval aggregate --input-glob override #4771

Merged

7 tasks done

prokotg mentioned this pull request Oct 23, 2026

feat(orchestration): run exports after each benchmark of gym eval submit #4974

Merged

prokotg force-pushed the prokotg/eval-export-record-exporters branch from 88ecef4 to d9d5c32 Compare October 23, 2026 12:48

copy-pr-bot Bot temporarily deployed to public October 23, 2026 13:17 Inactive

copy-pr-bot Bot temporarily deployed to public October 23, 2026 13:19 Inactive

copy-pr-bot Bot temporarily deployed to public October 27, 2026 09:06 Inactive

copy-pr-bot Bot temporarily deployed to public October 27, 2026 09:14 Inactive

copy-pr-bot Bot temporarily deployed to public October 27, 2026 09:16 Inactive

copy-pr-bot Bot temporarily deployed to public October 27, 2026 09:17 Inactive

prokotg force-pushed the prokotg/eval-export-record-exporters branch from a742002 to 60b400b Compare October 27, 2026 11:21

copy-pr-bot Bot temporarily deployed to public October 27, 2026 11:22 Inactive

copy-pr-bot Bot temporarily deployed to public October 27, 2026 11:24 Inactive

prokotg force-pushed the prokotg/eval-export-record-exporters branch from 60b400b to 44ea31d Compare October 27, 2026 12:50

copy-pr-bot Bot temporarily deployed to public October 27, 2026 12:50 Inactive

AdamRajfer reviewed Oct 27, 2026

View reviewed changes

prokotg added 8 commits October 28, 2026 10:09

docs(eval): point the live exporter settings at gym eval export for f…

80e2dbb

…inished runs Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>

docs(eval): explain the post-run fields, the from_global_config overr…

c118a1e

…ide and the upload filter Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>

docs(eval): say that skip_exporters is set by commands in code, not b…

eb660a7

…y users Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>

fix(eval): fail a post-run export whose backend reports no run id ins…

5f5fb31

…tead of writing an empty marker Signed-off-by: Tomasz Grzegorzek <tgrzegorzek@nvidia.com>

svcnvidia-nemo-ci reviewed Oct 28, 2026

View reviewed changes

marta-sd approved these changes Oct 28, 2026

View reviewed changes

Conversation

prokotg commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

PR plan

Validation

Known gaps

Rollout evidence

Uh oh!

copy-pr-bot Bot commented Oct 22, 2026

Uh oh!

prokotg commented Oct 22, 2026

Uh oh!

prokotg commented Oct 22, 2026

Uh oh!

github-actions Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

svcnvidia-nemo-ci left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

prokotg commented Oct 23, 2026

Uh oh!

prokotg commented Oct 23, 2026

Uh oh!

AdamRajfer left a comment

Choose a reason for hiding this comment

Uh oh!

AdamRajfer Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

prokotg Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

AdamRajfer Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

prokotg Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

prokotg commented Oct 27, 2026

Uh oh!

prokotg commented Oct 28, 2026

Uh oh!

prokotg commented Oct 28, 2026

Uh oh!

prokotg commented Oct 28, 2026

Uh oh!

svcnvidia-nemo-ci left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

prokotg commented Oct 22, 2026 •

edited

Loading

github-actions Bot commented Oct 22, 2026 •

edited

Loading