[prokotg]

Summary

• gym eval export evalrdl +rollouts_jsonl_fpath=... +label= +tags= +meta.k=v +parallel= +wait= +timeout= +url= +requirement= uploads one finished run to EvalRDL. EvalRDLExporter is an IsolatedExporter (feat(eval): add IsolatedExporter to run an exporter's backend work in its own venv #4915), so the evalrdl SDK lives in its own venv.
• The directory the rollouts file is in is uploaded as it is: everything there is part of the run (sidecars, health reports, model-call captures, benchmark-specific files), and nothing is copied or picked out. A run missing aggregate metrics or the saved config fails before any venv is built.
• The worker calls Client().uploads.upload(...):

EvalRDL	Result
committed	succeeded
deduped	skipped (nothing new stored)
accepted (no verdict before the timeout)	failed, exit 3; with +wait=false it is succeeded and the upload token is in the result
IntakeRefused	failed, exit 4, with stage and reason
other SDK error, or not exactly one upload	failed, exit 1

• The SDK reads its token from $EVALRDL_TOKEN, $GITLAB_TOKEN or $GITLAB_PRIVATE_TOKEN, then glab (usually missing on Slurm, so set a variable); there is no login cache. No package index is hard-coded: use uv's settings (such as UV_EXTRA_INDEX_URL) with the service's own <service URL>/v1/client/simple/ index, since evalrdl is not on PyPI. requirement= pins the version. The environment is built once and the SDK is not upgraded; a ClientTooOld failure names the venv to delete or +requirement=evalrdl==VERSION.
• The worker is killed after +timeout plus an hour for the upload itself (just the hour with +wait=false). The run directory is taken with .absolute(), so a symlinked rollouts file does not move the upload to its target's directory, and an export is refused when the directory holds another rollouts.jsonl or *_rollouts.jsonl, because EvalRDL reads a directory as one run. A stored run that reports needs_binding logs a line.
• Isolated-exporter changes: a worker may report an exit code and an exporter may take its requirements from its config. The staleness check moved to a shared unexportable_reason. (The sys.path fix, which stops evalrdl.py shadowing the SDK, is in feat(eval): add IsolatedExporter to run an exporter's backend work in its own venv #4915.)
• New opt-in test switch --with_evalrdl_sdk, like --with_downloads. CLI reference updated.

Part of #2977 (export-aware evaluation runs).

PR plan

Separate PRs; steps 1-8 are stacked (stack #4910). Goal: make gym eval export pluggable so a finished run can go to tracking systems without live exporters, and gym eval submit can export after each benchmark.

1. save the masked, unresolved config from gym eval run (feat(eval): save the masked config beside rollouts from gym eval run #4908)
2. RunArtifacts and path derivation helpers (feat(eval): add RunArtifacts to derive a run's files from its rollouts path #4909)
3. Exporter base class and registry, starting with atif, plus the positional exporter CLI argument (--format deprecated) (feat(eval): add the Exporter base class and registry, with atif as the first exporter #4911)
4. post-run mlflow and wandb export: shared metrics flattening, BaseExporter.export() and the already-exported guard (feat(eval): export a finished run to MLflow or W&B with gym eval export #4914)
5. isolated-venv subprocess runner for directory exporters (feat(eval): add IsolatedExporter to run an exporter's backend work in its own venv #4915)
6. This PR: EvalRDL exporter
7. gym eval submit integration: an export map with job-level defaults and per-benchmark overrides, validation, install extras, and a job-script tail (Slurm only at first) (feat(orchestration): run exports after each benchmark of gym eval submit #4974)
8. contributor guide: add an exporter (user-facing docs land in each step's PR) (docs(contribute): add a guide to writing an exporter for gym eval export #5077)

Validation

• New tests run the real worker as a subprocess against a fake evalrdl package with the SDK's names and signatures: config parsing, the run directory being uploaded as it is, every setting reaching the SDK, each state and exit code, refusal and accepted messages, run detail, wait=false, needs_binding, ClientTooOld and a missing token, a symlinked rollouts file, a directory holding another run's rollouts, and the worker time limit. 32 tests pass (1 opt-in skipped), and each of the new ones fails without its fix. pre-commit: passed.
• Real CLI with tags=[a,b] meta.k=v ...: all arguments parsed and the exporter reached the venv build, where it reported a clean failed (no SDK index here).
• The opt-in test_the_real_sdk_still_has_the_interface_the_worker_calls (pytest --with_evalrdl_sdk) checks upload() still takes the arguments the worker passes. It has NOT been run against the real SDK, only to confirm the flag works.

Manual test plan

End-to-end check of the atif, mlflow and evalrdl exporters on one small run. Run it from the top of this stack. I have not run the prepare, run or real-model export steps myself, because they need network access to HuggingFace and a model key (and, for EvalRDL, the SDK and its package index); the exporters themselves are covered by the tests above.

aime24 is used because it is small and needs no judge model (should_use_judge: false). It exercises the MLflow and EvalRDL exporters; ATIF needs a different input, see its subsection.

Prepare dataset

For a benchmark with a prepare_script, end-to-end gym eval run does not prepare the data, so this step is required. It needs network access.

source .venv/bin/activate        # policy_* settings in env.yaml
mkdir -p /tmp/gx
R=/tmp/gx/aime24_rollouts.jsonl

gym eval prepare --benchmark aime24
# writes benchmarks/aime24/data/aime24_benchmark.jsonl

Run

One command starts the servers, collects the rollouts and shuts the servers down.

gym eval run --benchmark aime24 --model-type openai_model --split benchmark \
    --output $R --limit 5 --num-repeats 1

ls /tmp/gx    # rollouts, _materialized_inputs, _aggregate_metrics.json, _resolved_config.yaml
grep -i -E "key|token" /tmp/gx/aime24_rollouts_resolved_config.yaml    # secrets show ****
# a URL with a password in a config value is saved as https://u:****@host/... (only the password is masked)

Exporter: ATIF

ATIF export is strict: it rejects any rollout whose ng_trajectory records a coverage gap. simple_agent (used by aime24) does not publish semantic turns, so its rollouts carry a turns_unavailable gap and are rejected by design: failed, exit 1, nothing written. A real run converts only when its agent publishes turns (for example scicode_agent, verifiers_agent or terminus_2_sandboxed_agent) and rollout evidence is enabled (+observability_enabled=true +model_call_capture_dir=<dir> on gym eval run).

Model-free check with a canonical sample run, from the Gym checkout (uses .venv because the helper imports pytest):

.venv/bin/python - <<'EOF'
import json, sys
sys.path.insert(0, "tests/unit_tests")
import test_atif_export as t
open("/tmp/gx/atif_sample.jsonl", "w").write("".join(json.dumps(t._canonical_rollout(task_index=i)) + "\n" for i in range(3)))
EOF
gym eval export atif +rollouts_jsonl_fpath=/tmp/gx/atif_sample.jsonl +output_dirpath=/tmp/gx/atif_sample +session_id=sample-1 +agent_version=0.0.1

• Expect succeeded, trajectory_count: 3 and a manifest.jsonl.
• Running it again fails with Output path already exists (exit 1).
• Optional: gym eval export atif +rollouts_jsonl_fpath=$R ... on the aime24 run fails with source coverage gaps are not supported by strict export on every row.

Exporter: MLflow

Use a tracking server, not MLflow's local file store: the file store turns each / in a metric name into a folder, so it cannot hold both <agent>/<metric>/no_answer and <agent>/<metric>/no_answer/std_dev_across_runs, which aime24 produces. A local server runs in its own throwaway environment; the first start downloads MLflow and takes about a minute. For a shared server, use its URL as mlflow_tracking_uri and put mlflow_tracking_token in env.yaml.

# terminal A: local server (port 5055 because macOS often reserves 5000)
uvx mlflow server --host 127.0.0.1 --port 5055 \
    --backend-store-uri sqlite:////tmp/gx/mlflow.db --default-artifact-root /tmp/gx/mlartifacts

# terminal B
gym eval export mlflow +rollouts_jsonl_fpath=$R \
    +mlflow_tracking_uri=http://127.0.0.1:5055 +mlflow_experiment_name=gym-export-test +mlflow_run_name=aime-1

• Expect succeeded and a run id. In the UI (http://127.0.0.1:5055): <agent>/mean/reward and key_metrics/... metrics, params from the saved config, and the global_config.json and rollouts.json artifacts.
• The same command again prints skipped (already exported) and creates no new run. Delete <stem>_mlflow_export_done to export again; add +upload_rollouts=false for a run without rollouts.json.
• Optional, live then export: run gym eval run with +mlflow_tracking_uri=... +mlflow_experiment_name=... +mlflow_run_name=... into the same output, then export. Expect a second run, a warning that names the live run id, and live_run_id in the result (the export is not skipped).
• Negative: move _aggregate_metrics.json away and export, expecting failed: Aggregate metrics not found and no marker file. A run without _resolved_config.yaml fails with Saved run config not found.

Exporter: EvalRDL

The package index is the EvalRDL service's own (<service URL>/v1/client/simple/), set through your uv settings. The token comes from $EVALRDL_TOKEN, $GITLAB_TOKEN, $GITLAB_PRIVATE_TOKEN or glab; evalrdl login or evalrdl doctor verify it.

export UV_EXTRA_INDEX_URL=<your organization's package index>
gym eval export evalrdl +rollouts_jsonl_fpath=$R +label=gym-export-smoke '+tags=[smoke]' +meta.source=manual +url=stg

• Expect succeeded with a run id and upload token; the first run builds the exporter environment.
• A second run reports skipped (deduped) with exit 0.
• With +wait=false, expect succeeded and an upload token without waiting for a verdict.
• Exit code 3 means the upload was stored without a verdict before the timeout (only when waiting); exit code 4 means it was refused (the message names the stage and reason).
• Negative: the same missing-aggregate-metrics and missing-saved-config checks as for MLflow fail before any environment is built. A second *rollouts.jsonl in the run's directory fails with a message saying to keep each run in a directory of its own. With no token set (and no glab), expect a failed result that names the three token variables.
• Opt-in SDK contract test: pytest tests/unit_tests/test_evalrdl_exporter.py -k real_sdk --with_evalrdl_sdk.

Known gaps

• SDK usage follows the published reference; it has not been exercised against the real service.
• Everything in the rollouts' directory is uploaded, so each run should have a directory of its own (the export is refused when another run's rollouts file is there).
• If the SDK splits the directory into several child uploads, Gym's top-level files are not uploaded and a single child can still report succeeded. Detecting that needs upload_plan (not in the released SDK) or a private SDK function, so it is not checked.

Rollout evidence

N/A: no agent, verifier or model behavior is touched.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[prokotg]

/review

[prokotg]

/ok to test 5228263

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4965-8f42e3fa859d.docs.buildwithfern.com/nemo/gym

[svcnvidia-nemo-ci]

SHIP WITH CARE — one RISK, no BLOCKER findings.

Reviewed all 10 changed files and their directly affected CLI, artifact, and exporter contracts. Expected filesystem failures during staging currently escape instead of returning a failed export result.

Validation was static only; no tests, SDK installation, or service calls were executed.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

[prokotg]

/ok to test c31638b

[prokotg]

/ok to test 8f42e3f

[prokotg]

/review

[svcnvidia-nemo-ci]

SHIP — no remaining eligible findings in this follow-up.

The prior staging-copy RISK is resolved: build_payload no longer stages or copies files, eliminating the reported copy2 failure path. The current regression tests cover directory selection, symlink handling, neighboring-run rejection, and prerequisite failures.

Checked the identifiable subsequent changes and directly affected paths: accepted uploads without waiting, preserved wait/refusal exit codes, diagnostics, worker time limits, isolated-environment remediation, and associated documentation/tests. No new regression qualifying above the historical moderate-impact ceiling was established.

This was a follow-up, not a fresh whole-PR review. The previously reviewed tree was unavailable, limiting direct inter-review comparison. Validation was static only; no tests, SDK installation, or service calls were executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

[agronskiy]

Review by Alex Gronskiy (EvalRDL side). All addressed, thanks. I checked the current head; the --with_evalrdl_sdk check passes against the released SDK 0.0.241. Approving so the train can merge.

Follow-up on the split check: the reason it was blocked no longer holds. SDK 0.0.241 (released yesterday) exports a public upload_plan from evalrdl.sdk.api. I reproduced the problem with it: a directory holding a_rollouts.jsonl plus other/logs/x.json is planned as one upload of other/ only, so the worker would report succeeded without uploading the Gym run. Suggested fix, fine as a separate PR:

• in the worker, before upload(): plans = upload_plan(run_dir), and fail unless len(plans) == 1 and plans[0]["dir"] == run_dir;
• default requirement to evalrdl>=0.0.241.

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40689158747