Repository navigation
feat(exporters): keep a resumed or resubmitted run in one MLflow run - #4762
Merged
AdamRajfer merged 2 commits intoOct 20, 2026
Merged
Conversation
Every job of an evaluation created its own MLflow run, so a run that auto-resumed or was resubmitted was split across MLflow runs with the same name, each with its own metrics. The exporter now writes the run id to <output stem>_mlflow_run_id next to output_jsonl_fpath and reopens that run when the same output starts again. Deleting the file starts a new MLflow run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-4762-95d75cc0a69d.docs.buildwithfern.com/nemo/gym |
yaoyu-33
added
area:evaluation
marta-sd
reviewed
Oct 20, 2026
…kend The run id file moves from the MLflow exporter into BaseExporter, so every backend keeps a resumed or resubmitted run in one run the same way: the id is written to <output stem>_<backend>_run_id next to output_jsonl_fpath. The MLflow file name and behaviour are unchanged. The W&B exporter now records its run id too and resumes that run when the same output starts again. Deleting the file starts a new run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
marta-sd
approved these changes
Oct 20, 2026
Contributor
Author
|
/ok to test 95d75cc |
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39416802760 |
jin-nvidia
pushed a commit
to cparisien/Gym
that referenced
this pull request
Oct 27, 2026
…nded rollout once (NVIDIA-NeMo#5029) * fix(exporters): reopen a tracking run only when the run resumes, and log each rollout once NVIDIA-NeMo#4762 reopens the MLflow or W&B run recorded next to an output, so that a resumed or resubmitted run stays one tracking run. Three cases went wrong: - A fresh run into an output that held an earlier run reopened that run, merging two evaluations. A run now reopens only when it resumes its output (resume_from_cache); otherwise it starts a new run. - A resumed collection hands the exporters every rollout of its output, and MLflow's log_table appends, so a reopened run's table held the earlier rollouts twice. A reopened MLflow run now logs only rollouts its table does not hold yet. - A deleted MLflow run could not be reopened, so logging stopped for that output. A new run now takes its place. The new tests run against a real local MLflow store. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(exporters): log a rollout that lands on resume after a failed attempt A reopened MLflow run skipped every rollout whose task and rollout index its table already held, failed attempts included. A rollout that failed in one job and landed on resume therefore kept only its failure record in the table. Now only the landed rollouts the table holds are skipped; a failed attempt stays in the table as it was logged, next to the later landing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(exporters): reopen the tracking run on a resubmit too A resubmitted job adds only resubmit=True to its config. Collection turns that into a resume, but the exporters open earlier, while the global config is parsed. The reopen gate therefore saw no resume and started a new MLflow or W&B run for the job's continuation. A resubmit now reopens the recorded run, as a resume does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(exporters): keep the tracking run in step with what collection continues The reopen gate and the landed-rollout filter got three cases wrong: - A no-persist attempt is uploaded but written nowhere, yet it counted as landed, so a rollout that landed later for the same task was left out of the MLflow table. A rollout counts as landed now only without a failure class and without the no-persist flag. - Judge-failure recovery in append mode adds its rows to the output it recovers, but started a new tracking run. It reopens the recorded run now, as a resume does. - A resume that finds nothing to resume starts its output anew, after the exporters reopened the recorded run, so the MLflow filter dropped fresh rollouts whose keys were in the old table. Collection now tells the exporters: they leave the old run as the earlier job left it and track the output in a new run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(exporters): let go of the reopened run even when restoring it fails Starting over cleared the exporter's handle on the reopened run only after restoring that run's status and resume count, or after finishing the W&B run. When the tracking server refused that, the guard swallowed the error before a new run opened, and the fresh output kept logging to the old run, where the MLflow filter dropped its rollouts as landed. Each backend now lets go of the reopened run first, and restoring it is best effort: a failure is logged and the new run still opens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(exporters): keep restarted output in its replacement run Defer a reopened run configuration until continuation is known, clear its marker before replacement setup, and start a fresh tracking run when reverification finds no cached output. Cover dispatcher ordering, saved answers, masked configuration and best-effort cleanup with regressions. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(exporters): parse continuation flags consistently Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(orchestration): normalize resume before exporter setup Resubmitted Slurm jobs set resume_from_cache before exporters open. Normalize the resubmit command in older saved job scripts under the existing lock, preserving their tracking identity and rollout artifacts. Exporters now use only resume and append-recovery flags. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> --------- Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every job of an evaluation created its own MLflow run, so a run that auto-resumed or was resubmitted was split across MLflow runs with the same name, each with its own metrics.
Changes:
<output stem>_mlflow_run_idnext tooutput_jsonl_fpath, and reopens that run when the same output starts again.Tested with a new unit test in
tests/unit_tests/test_exporters.py: a second exporter on the same output reopens the recorded run instead of creating a new one.🤖 Generated with Claude Code