[marcromeyn]

Scope: diagnostic capture only. Retry and recovery policy stay in #5072/#5073/#4996, and #3302's run manifest keeps its own interpretation of never-started, interrupted and omitted work. Reviewed with @AdamRajfer; his two findings are fixed in 7fa409f: readers keep the latest execution visible before setting these rows aside, and key on their own marker (_ng_omitted_rollout), so existing ATIF rows are read as before.

What and why

Today a rollout that produced no result leaves no row in any artifact:

• an attempt the agent reports as kill_shaped (_ng_no_persist: true; Slurm SIGTERM, a dead Ray actor, an OOM
  outside the rollout), and
• a row dispatch_budget_s never started (_ng_dispatch_drained, class cancelled).

Resume relies on that: no row means "run it again". But absence is then the only evidence. The materialized inputs
show the rollout was owed. Nothing says it was killed or never started, or why.

This is the "no-result attempt" half of GYM-327 ("define typed rollout outcomes and recovery"), which notes that
kill-shaped omissions leave no evidence. It is offered as an input to that work, not a separate design. It adds no
outcome taxonomy and no new failure class, and one new key, _ng_omitted_rollout, that marks the rows it writes. It
writes down what the collector already knows, using the markers the result already carries. GYM-327 can later map these rows onto a typed outcome. For example
nemo_gym/rollout_outcomes.py's RolloutFailure (not yet used by the collector) has source="collector" and
delivery="not_sent" for a drained row and "possibly_delivered" for a killed attempt. This PR does not decide
that mapping.

Writing (nemo_gym/omitted_rollouts.py, called from rollout_collection.py): with
+record_omitted_rollouts=true, each _ng_no_persist result is appended to <stem>_failures.jsonl as one row:

• a copy of the result, so _ng_task_index, _ng_rollout_index, agent_ref, _ng_attempt_index (on a retry) and
  any diagnostics the agent returned stay with it;
• _ng_no_persist: true is kept, and _ng_omitted_rollout: true is added: readers key on the latter, because ATIF
  reverification already writes _ng_no_persist on rows it counts as failures;
• _ng_failure_class: the result's own class. If the result has none, cancelled for a drained row, otherwise
  kill_shaped;
• _ng_failure_message: the reason the result gave (_ng_failure_message, error_message or error). Without one it
  is "<class>: the rollout produced no result". A drained row says, for example,
  not dispatched: dispatch budget exhausted.

The rows are not attempts: no reward is added, no _ng_failure_max_attempts is stamped, and they never enter
failure_counts.

Reading (unconditional, so a sidecar written with the option is read correctly by a run without it). A row is
never a failure or a failed attempt. Where a reader takes a rollout's latest attempt, an interrupted row
(kill_shaped) is that latest execution, so an older failure is not counted again and the rollout is missing; a row
the dispatch budget never started (_ng_dispatch_drained) never advances history. Rows with only _ng_no_persist
are read exactly as on main.

Reader	File	Effect
scoring, count_failure_classes_as_zero, batch status, gym eval aggregate (_latest_failure_rows)	rollout_collection.py	latest execution first, then not a failure; count_missing_rollouts_as_zero counts the rollout as missing
exporter metrics (_failure_export_metrics)	rollout_collection.py	no retries/attempts; an interrupted latest execution hides an older coverage/failed/<class>
resume (_load_from_cache)	rollout_collection.py	no attempt counted, _ng_failure_terminal ignored, the rollout runs again unstamped (retry policy unchanged)
run health (_rollout_accounting)	rollout_health.py	latest execution first; the rollout is missing, not failed
gym dev compare (load_rollouts)	rollout_compare/rows.py	latest execution first; no masked failure row, the rollout is missing
gym eval reverify --judge-failed-only	rollout_reverification.py	not a judge failure to recover, and does not trigger the multi-stage refusal

gym eval reverify --judge-failed-only still selects judge failures as on main; how a later execution affects
rejudging a saved answer belongs to the recovery work (#5073, #3879).

Why the option is off by default

EvalRDL's nemo-gym reader counts every failures-sidecar row of an unfinished rollout as a failed rollout. If this
were on by default, every run with a kill-shaped attempt or a drained budget would show "not run" rollouts as failures
in EvalRDL. That is the same missing-reported-as-failed mistake this change exists to avoid. A matching EvalRDL change
(read these rows as not run, run.n_not_run) is in review separately. Even after it merges, an
older EvalRDL deployment reading newer Gym output would still miscount, so the default should flip only once that
reader has shipped. Until then the cost is one boolean: Gym's own readers already handle the rows, and the option can
be removed when the default flips. The option does not depend on that change. It can be turned on today for runs
whose sidecar is read only by Gym.

Validation

All commands from the repository root.

python -m pytest -q -p no:cacheprovider -o addopts="" -n 8 \
  tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_rollout_reverification.py \
  tests/unit_tests/test_rollout_health.py tests/unit_tests/test_compare.py tests/unit_tests/test_dev_compare.py \
  tests/unit_tests/test_batch_status.py resources_servers/gdpval/tests/test_multistage_orchestrator.py \
  tests/unit_tests/test_fern_docs_links.py

1350 passed, 24 subtests passed at 985b9ef (7fa409f merged with current main), with the telemetry extra installed (without it
test_rollout_collection.py is skipped as a whole).

New and extended tests (20 new test items):

• TestOmittedRolloutRecords (5 cases, table-driven): killed; killed with no class or reason; no result under another
  class; budget exhausted; below the drain margin. Each case runs run_from_config and checks for exactly one marked
  row, with its class, reason, identity and no added reward. It then resumes, and checks that the rollout is sent
  again with no attempt index and lands in the rollouts JSONL.
• TestSidecarReadersSkipOmittedRows (5): scoring and missing-as-zero, exporter metrics, run health,
  gym dev compare, and resume (five terminal copies of a marked row do not retire the rollout).
• Existing end-to-end tests gain an "omitted recorded" parameter, and the existing assertions still hold:
  • test_run_from_config_counts_a_rollout_that_produced_no_row_as_zero: the score is the same, with
    count_failure_classes_as_zero=[kill_shaped] set;
  • test_rows_the_budget_never_started_count_as_zero_until_resume_runs_them: the score and the resume are the same;
  • test_run_from_config_non_retaining_preserves_upload_semantics: the upload is the same, including on resume over
    a sidecar that already holds a marked row.
• test_rows_for_rollouts_with_no_result_are_not_recovered (4) in test_rollout_reverification.py.
• TestOmittedRowsAndTheLatestAttempt (4, after review): in scoring, exporter metrics, run health and
  gym dev compare, an older judge_failed followed by an interrupted row is not counted again (the rollout is
  missing), an older judge_failed followed by a never-started row still stands, and an ATIF-style row with only
  _ng_no_persist is still counted. All four fail on the previous head (958e1e6).

Red/green: with only the production files stashed, the selected tests fail 17 times. Three new items pass without
the production change and are kept as regression guards: the upload case (the option adds nothing to uploads) and the
reverification killed and never-started cases (their classes were never recovered).

The full tests/unit_tests/ run and the red/green check above were done before rebasing onto current main and have not been repeated.

pre-commit run --files <changed files> passes on the rebased commit. The guardwords hook skipped
itself because the private catalog is not available.

Rollout evidence

On the head before rebasing onto current main and before review; the rows now also carry _ng_omitted_rollout.
Real run: example_single_tool_call + simple_agent against openai_model (gpt-6-luna), 3 tasks,
--concurrency 1 --max-output-tokens 512 +dispatch_budget_s=2. Task 0 started. By the time it finished, the 2 s
budget was spent, so tasks 1 and 2 were never started.

Run	Options	rollouts.jsonl	rollouts_failures.jsonl	Health (quality_summary.json)
default	budget only	task 0, reward 1.0	empty (0 bytes)	n/a
recorded	+record_omitted_rollouts=true	task 0, reward 1.0	2 rows: tasks 1, 2	expected 3, failed 0, missing 2
recorded, --resume	same, no budget	tasks 0, 1, 2, all reward 1.0	the same 2 rows	n/a

Each recorded row: _ng_no_persist: true, _ng_dispatch_drained: true, _ng_failure_class: "cancelled",
_ng_failure_message: "not dispatched: dispatch budget exhausted", agent_ref, task and rollout index, no reward.

• Metrics. mean/reward was 1.0 over 1 rollout. The completion report said "Rollouts missing from the score: 2 of
  3" with no failures routed.
• Resume. The resume printed 0 prior failure attempts, 2 rows that still need to be run, and ran both.

A kill-shaped attempt was not run against a real model: the example agent cannot be made to report kill_shaped from
configuration. The killed cases above cover it with a stubbed agent reply.

Compatibility

• Default artifacts are unchanged. With the option off, nothing new is written.
• Existing _ng_no_persist rows are read as before. ATIF reverification's kill-shaped rows carry no
  _ng_omitted_rollout, so their failure counts and requested kill_shaped zeros are unchanged.
• Rows can carry the agent's own fields. A recorded row is a copy of the result. An agent that puts a placeholder
  reward on a kill-shaped result (pinchbench writes reward: 0.0) keeps it in the row. External readers must check
  _ng_no_persist before reading reward.
• Uploads are unchanged. Kill-shaped results were already uploaded and still are. Drained rows were not uploaded
  and still are not. The sidecar row is never uploaded.
• Not covered.
  • A custom rollout_collection_driver ignores the option. GDPVal's multi-stage driver keeps its own routing and
    writes nothing for _ng_no_persist rows.
  • A driver that is itself killed before it sees an outcome still leaves no row.
• Scores change only with the option on, and only where a count_*_as_zero option is set. A rollout whose latest
  execution was interrupted is then missing rather than counted under an older failure class. With the option off
  (the default), no artifact, metric or score changes.

Docs: reference/cli-commands.mdx (a "Rollouts with no result" bullet, plus "Counting lost rollouts" and "Dispatch
budget" wording) and evaluation/aggregate-metrics.mdx.

Proposed labels: area:evaluation.

Linear: FEP-1842 (part of FEP-1829)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[AdamRajfer]

The diagnostic capture scope makes sense. Two reader behaviors need attention before merge.

[marcromeyn]

/ok to test 985b9ef

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5081-41ab898cc688.docs.buildwithfern.com/nemo/gym

[marcromeyn]

/ok to test 41ab898

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40394147664

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40394677434