[GDPVal] Harden production rollout and judging paths by Kh4L · Pull Request #2588 · NVIDIA-NeMo/Gym

Kh4L · 2026-08-18T02:40:58Z

Summary

Route malformed, truncated, or scoreless judge responses out of headline metrics and through retry-aware failure sidecars; preserve completed policy artifacts for re-judging and make deterministic payload/context failures terminal.
Bound GDPVal text, binary, Office, PDF, media, and ZIP handling before reads and request construction; preserve sidecar provenance, attachment fairness, and A/B swap invariance while avoiding repeated hashing for payloads that already fit.
Make standard and multi-stage collection honor finite dispatch budgets, stage-aware longest-first scheduling, attempt accounting, and cached-deliverable reuse; harden resume with config fingerprints, legacy migration, completion coverage checks, atomic writes, and stale-metric cleanup.
Allow opt-in bounded partial completion for non-final calibration stages after persisted timeouts, with overall and per-reference evidence floors, balanced fresh assignments, frozen evidence snapshots, and fail-closed resume. The final stage still requires full evidence for a headline ELO.
Sanitize unparsed raw tool-call blocks before provider requests and guarantee globally unique converted tool-call IDs without mutating stored trajectories.
Align reverification with the new judge-invalid failure contract, preserve file/reference context, migrate legacy invalid rows, and reject unsafe multi-stage judge-failure recovery.

Behavior changes

dispatch_budget_s now requires a finite positive num_samples_in_parallel.
Missing rubrics and deterministic request-size/context failures are terminal instead of consuming repeated judge attempts.
--judge-failed-only fails fast for multi-stage sidecars; multi-stage collection resume is the supported recovery path for those rows.
partial_completion is opt-in and valid only for non-final stages. Resume uses the same output path plus --resume; changing a policy after a partial outcome has been frozen fails closed.

Validation

env PYTHONPATH=. uv run pytest resources_servers/gdpval/tests -q
- 310 passed, 9 optional-dependency skips
env PYTHONPATH=. uv run pytest tests/unit_tests/test_deliverables.py tests/unit_tests/test_dispatch_latency.py tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_rollout_reverification.py -q
- 229 passed
Focused multi-stage suite, including 41/45 calibration recovery, interrupted-final-stage resume, stale downstream invalidation, frozen-evidence validation, and crash-order handling:
- 135 passed
Ruff check and format check pass for every changed Python file.
git diff --check passes.

copy-pr-bot · 2026-08-18T02:41:02Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Serge Panev <spanev@nvidia.com>

agronskiy · 2026-08-21T13:58:08Z

@Kh4L it seems for me, the logic prevents timing out-tasks (the ones that could not finish in 3.5 hours cut-off) of stage-1 prevent starting stage-2. I think we should admit that some GDPVal tasks will timeout on stage-1 as either hard or smth like that -- and this should not prevent apporximately calibrating and starting stage-2. WDYT?

Signed-off-by: Serge Panev <spanev@nvidia.com>

Kh4L · 2026-08-22T00:44:30Z

Implemented the Stage 1 timeout recovery in 14125b765.

A non-final stage can now opt into bounded partial completion with overall and per-reference coverage floors.
An existing 41/45 Stage 1 run with four persisted timeouts can advance on --resume without redispatching those four long-running tasks.
Fresh policy-enabled stages balance task assignments across the selected references.
The accepted evidence and omitted keys are frozen, so later resume cannot silently change Stage 2 reference selection.
The final stage remains strict: an incomplete final stage emits no headline ELO and resume dispatches only its missing rows.

The recovery configuration and --resume/RERUN_INCOMPLETE distinction are documented in benchmarks/gdpval/README.md. The exact 41/45 → partial calibration → interrupted final stage → single-row final resume path is covered by the focused suite.

…le (#2704) On top of #2588, `PartialStagePolicy.waivable_failure_classes` becomes a configurable tuple instead of being hardcoded to `("timeout_exceeded",)`. The default is unchanged, so existing configs behave identically. ## Change - Commit: `36767c8052c76c7047551262e19fb1afd481233e`, branched off `14125b7650317b1d1269987c840ea70deba7e5f4` (the tip of #2588 at the time). - Files touched: `resources_servers/gdpval/multistage_elo.py` (new field), `resources_servers/gdpval/multistage_orchestrator.py` (7 edits: policy record now emits `newly_waivable_failure_classes`; the waiver check consults the configured set; `_WAIVABLE_FAILURE_CLASSES` validation frozenset is `{"timeout_exceeded", "transient"}`; snapshot revalidation maps the record key `newly_waivable_failure_classes` back to the config key `waivable_failure_classes`), plus 3 new tests in `resources_servers/gdpval/tests/test_multistage_orchestrator.py` (`test_transient_omission_advances_when_explicitly_waivable`, `test_unknown_waivable_failure_class_is_rejected`, `test_empty_waivable_failure_classes_is_rejected`). - Test status: 313 passed, 9 skipped across `resources_servers/gdpval/tests/`. ## Motivation The operative blocker was NOT the waiver scope: `partial_completion` is opt-in and was never set, so `multistage_orchestrator.py:1090` took the strict branch and required every planned non-final rollout to carry battle evidence. That cost two GDPVal-AA-v2 runs their entire stage 2 (`dc6c776f3af506df` and `6c5ee8b81773b2b6`, 44/220 each, `eval_elo` null). Once `partial_completion` is enabled, a `transient` judge failure still stopped the stage because the waiver set was hardcoded — which this PR fixes. ## Production evidence From Nemotron Super 3.5 run `590aeb2c72b1de1d` on this commit: - stage 1 log line: `[multistage-elo] stage 1/2 accepted partial calibration: success coverage 91.1%, omitted 4 rollout(s)` - omitted rollouts and classes: task 78 `timeout_exceeded`, task 155 `timeout_exceeded`, task 159 `timeout_exceeded`, task 136 `transient` - `success_fraction` 0.9111, worst per-reference 0.8, `multistage_state` stage-0 status `partial_complete` - stage 2 then ran to completion: `eval ELO = 737.5` over 4 refs. The predecessor run died at 44/220 with `eval_elo` null. - The run also spanned a slurm TIMEOUT resume, and the partial stage was correctly reloaded: `[multistage-elo] resuming multi-stage run from cache (fingerprint match)` and `[multistage-elo] stage 1/2 reused partial result from cache: eval ELO = 627.4 (41 cached rollout(s))` — so the `newly_waivable_failure_classes` <-> `waivable_failure_classes` round-trip works across legs. - A second run on the same commit, Qwen3.5-122B-A10B `df318339a24e9384`, completed at `eval ELO = 882.6` with a clean 45/45 stage-1 close (the waiver path did not fire there). ## Pre-merge validation Against `36767c80`, with policy `{min_success_fraction: 0.9, min_per_reference_success_fraction: 0.5, min_successful_rows_per_reference: 1, waivable_failure_classes: [timeout_exceeded, transient]}` and a synthetic 45-rollout stage: an omission of class `timeout_exceeded` accepts at `success_fraction` 0.9778, `transient` accepts at 0.9778, and `legitimate` is rejected. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…re (#2796) A comparison verify against a reference model that has **no deliverable for the task** returned `reward: 0.0` with `judge_response: {"error": "reference_missing"}` and no failure class, so the harness persisted it as an ordinary **success row**. A success row carrying no battle evidence is rejected outright by the non-final-stage coverage gate in `_partial_stage_outcome` (`if set(successful_by_key) - fit_eligible_keys: return None`), which runs *before any threshold is consulted*; the waiver machinery never sees the row either, since `waivable_failure_classes` only applies to rows that failed. Net effect: one missing reference file killed an entire multi-stage run, no `partial_completion` setting could relax it, and the run reported a stage incomplete while every configured gate had passed. This PR stamps the response with `_ng_failure_class=reference_missing` and marks it terminal — retrying cannot make the file appear, the same reasoning already applied to `skipped`. The row then leaves the success set, becomes visible in the failures sidecar, and is governed by the stage's `partial_completion` fractions like any other unusable row. It also sets `extra="allow"` on `GDPValVerifyResponse`: the underscore-prefixed harness keys cannot be declared as pydantic fields, and the default `extra="ignore"` dropped them **silently**, so a response that tried to stamp a failure class was serialised without one. Observed three times on GDPVal-AA-v2 in two days — DeepSeek V4 Flash (`9e39df84` / `qwen35_397b`), Inkling-Small (`1752cb53` / `qwen35_397b`), Nemotron 3 Ultra (`e222075d` / `qwen36_35b`) — where the first two lost stage 2 entirely at 97.8% and 95.6% coverage and the third survived only because it landed in the final stage, which is never coverage-gated. This is a **mitigation, not a cure**: the underlying defect is incomplete reference deliverable sets (`qwen35_397b` 212/220, `qwen36_35b` 216/220, `gptoss_120b` 218/220 — 14 missing pairs), and backfilling those removes the cause. A reviewer may reasonably prefer the stricter structural fix of refusing to *plan* a (task, reference) pair whose reference deliverable is absent, so a run never draws one; that would live in the task-distribution builder and is not attempted here. Tests: `test_verify_comparison_missing_reference` and `test_reference_ids_empty_yields_no_references` extended to assert both keys survive `model_dump()`; new `TestReferenceMissingCoverage` (terminal omission accepted without being waivable, still rejected below `min_success_fraction`, and an evidence-less success row still hard-rejects — the gate is not weakened) and `TestReferenceMissingIsTerminal`. Diff is confined to `resources_servers/gdpval/`; CI green. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>

Add an opt-in strict comparison contract for reportable ELO runs. Reject missing artifacts, failed matchups, invalid verdicts, and incomplete vote panels before results enter the resume cache. Keep legacy behavior as the default and fingerprint strict runs separately. ## Checklist - [X] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [ ] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [ ] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [ ] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [ ] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). Signed-off-by: Serge Panev <spanev@nvidia.com>

…lity replay (#2896) Ports the per-provider judge transport layer, executed and validated in a full 220-task GDPVal evaluation campaign, onto the production-hardening branch. ## Changes - Per-judge media transport modes: `native_pdf`, `images_and_text`, and `native_pdf_overflow_images` (native PDFs kept; over-cap PDFs rasterized at a pinned DPI with every page retained). - Provider-specific request limits as judge-panel config: serialized-request bytes, native-PDF page/document/byte caps (aggregate and per-document), video count, raster DPI tiers. - Per-document native-PDF cap (`max_native_pdf_bytes_per_document`) forces lossless full rasterization of any PDF above the provider's per-file limit; page-count overflow rasterizes only the required prefix, keeping the suffix native. - Deterministic transport eligibility preflight before any billable call, with seeded schedule replay; exclusions recorded per row (`media_routing_exclusions`, `transport_by_judge`). - Overflow eligibility also enforces the judge's aggregate document/byte caps against the post-overflow native remainder, and a render page cap so a plan the renderer cannot satisfy is excluded at preflight instead of failing at dispatch. - New `transport_assignment.py`: provider-free task/reference reassignment repair using exact artifact footprints against provider caps. - `transport_assignment_repair` settings enter the multistage fingerprint, so changing them invalidates journals instead of silently replaying stale plans on resume. - Final verdict reminder block appended to judge messages; the text budget's framing reserve includes its serialized upper bound so the request-size invariant holds at any configured cap. ## Validation - `resources_servers/gdpval/tests`: 379 passed, 3 skipped, including 16 new tests in `test_transport_modes.py` (overflow planning/application, eligibility receipts, seeded replay, assignment solver capacity contract, fingerprint invalidation). - `ruff check` / `ruff format` clean. - Same code path that produced a strict 220/220-task, 880/880-trial judging PASS in the validating campaign, rebased onto this branch. First of four planned child PRs (media caps/tiling, provider timeout/retry hardening, transport receipts follow). Draft until the example three-provider judge panel config lands. --------- Signed-off-by: Serge Panev <spanev@nvidia.com>

…eligibility Signed-off-by: Serge Panev <spanev@nvidia.com>

…nt runtime (NVIDIA-NeMo#3685) ## What does this PR do? This PR moves the Stirrup sandbox and runtime work from `gdpval-devel` to main. On devel it landed as NVIDIA-NeMo#2338, NVIDIA-NeMo#2395, NVIDIA-NeMo#2588, NVIDIA-NeMo#3086, NVIDIA-NeMo#3121 and NVIDIA-NeMo#3333 (sandbox part). It is one of a series of PRs cut by file from a single tested devel snapshot. It touches `responses_api_agents/stirrup_agent/` and the GDPVal sandbox build script in `nemotron_recipes/`. The GDPVal prompt rewrite that describes this sandbox is in NVIDIA-NeMo#3686, not here. **Sandbox (NVIDIA-NeMo#3333)** - `containers/gdpval.def` now builds the published GDPval-AA v2 environment: - `debian:trixie` with Debian's Python 3.13.5. The build fails on any other interpreter version. - The 419 published Python pins, installed verbatim and checked for exact versions afterwards. - Carlito/Caladea (metric-compatible Calibri/Cambria) and the Noto fonts. - Both published manifests are vendored as plain text: 419 Python pins and the 762-package apt closure. On arm64 the build drops exactly the 5 pins in `gdpval_aa_v2_arm64_exclusions.txt`. Any other pin that fails to install fails the build. - `verify_gdpval_sandbox.py` is staged into the image. It checks a built `.sif` against both manifests, then runs the toolchain end to end. - The definition stages its manifests with relative `%files` paths, so it has to be built from `containers/`. The README command is updated for that. `nemotron_recipes/.../build-gdpval-sif.sh` now builds from the definition's directory, where before it built from the repo root, which fails with this definition. - `_build_command_script` is pulled out of `ApptainerCodeExecToolProvider._exec` with no behaviour change. A test runs two provider-built commands in one bash and checks that only files persist between calls. **Model client and agent (NVIDIA-NeMo#2338, NVIDIA-NeMo#3121)** - `DynamicMaxTokensChatCompletionsClient`: - When the tokenizer renders the complete prompt, the remaining context also caps `max_completion_tokens`. A prompt with no room left raises Stirrup's `ContextOverflowError` before the request is sent. - For token counting only, tool-call arguments are JSON-decoded before template rendering, as vLLM does. - Without a tokenizer, or when tokenization fails, the character estimate counts the whole JSON-serialized history, tool calls included. - New arguments: - `min_completion_tokens`, default 1024 (the old constant). - `prompt_estimator_truncate_history_thinking`, default `None`. It only affects the estimate. - `truncation_recovery`, default `False`. When on, a turn that uses its whole budget without a schema-valid tool call is followed by one turn that requests thinking off and adds a one-time instruction. The instruction is not recorded in the trajectory. - `NeMoAgent.summarize_messages`: - Strips a leading `<think>` block from compaction summaries. - Makes up to 3 summary attempts. Retries add a stricter text-only instruction, and the last attempt runs without tools. If all 3 fail it raises instead of wiping the history. - New `min_compaction_summary_words` argument, default 1. **Provider payload and trajectories (NVIDIA-NeMo#2395, NVIDIA-NeMo#2588, NVIDIA-NeMo#3086)** - `to_provider_openai_messages` sends only the standard OpenAI message keys: - Stirrup's serializer also emits `reasoning_content`, `thinking_blocks` and `metadata` on assistant messages, flat `tool_call_id`/`name`/`arguments`/`signature` fields on tool calls, and `name` on tool messages. - Main's strict chat schema rejects those fields. So any multi-turn request whose history has an assistant turn with tool calls or reasoning currently fails against a Gym model server. - Unparsed `<tool_call>` / `<function=...>` blocks are removed from the provider copy of assistant text. This applies only to assistant turns with no parsed `tool_calls`, and an unclosed block is stripped to the end of the message. The recorded history keeps them. - `convert_stirrup_history_to_output_items` makes `call_id` unique within a response when the server reuses ids across turns (e.g. `code_exec:0`). Call/output pairing is preserved, and trajectories without repeated ids are unchanged. - Tavily `web_fetch`: - If markdown extraction is empty, it tries a second plain-text pass that includes comments. If that is also empty it returns an explicit error. - An extraction under 1% of a page of 20,000+ characters gets a "likely a fragment" note at the start. The new client and agent options are documented in the Stirrup README as constructor arguments. The agent config keys that set them come with NVIDIA-NeMo#3686, which owns `stirrup_agent/app.py`. **Issue:** none. This promotes work that was already reviewed and merged on `gdpval-devel`. **Dependencies:** - CI: none. This PR passes the Stirrup and GDPVal suites on main by itself. - Runtime, already true on main before this PR: main's strict chat schema returns 422 for the `chat_template_kwargs` that Stirrup's client sends on every call. Running Stirrup against a Gym vLLM model server therefore needs NVIDIA-NeMo#3682. - Thinking control: under NVIDIA-NeMo#3682's current default (`forward_request_chat_template_kwargs: false`), the model server accepts request `chat_template_kwargs` and drops them. That means Stirrup's `enable_thinking`, and the thinking-off half of `truncation_recovery`, reach vLLM only when the policy model server sets `forward_request_chat_template_kwargs: true`. The GDPVal config in NVIDIA-NeMo#3686 doesn't set it, so there only the recovery instruction takes effect. The default is still open on NVIDIA-NeMo#3682. - NVIDIA-NeMo#3686 (lands after this one): - It brings the GDPVal prompt rewrite (`code_exec`, no shell state between calls, the 5-minute limit, the AA v2 package summary). It also brings the reference-file path fix that prompt relies on, and the tests that tie the prompt to this sandbox's manifests. - It adds the agent config keys `context_window_tokens`, `min_completion_tokens`, `prompt_estimator_truncate_history_thinking`, `min_compaction_summary_words` and `truncation_recovery`. - Until it lands, main's prompt keeps advertising `run_shell` and packages the image doesn't have (torch, keras, jax, dlib). Main's current image doesn't have them either. The new options keep their defaults, and the context window still comes from the request's `max_output_tokens` (fallback 262144). ## Validation Main `1c8261080` + only this PR's 17 files, macOS, Python 3.13, per-server venvs: - `cd responses_api_agents/stirrup_agent && python -m pytest tests -q`: **224 passed, 2 skipped**. The skips need GNU `timeout` or a real Apptainer image (`GDPVAL_CONTAINER_PATH`). - `cd resources_servers/gdpval && python -m pytest tests -q` (gdpval imports Stirrup): **156 passed**. - With the rest of the series applied: Stirrup 319 passed / 2 skipped, GDPVal 564 passed. The 12 prompt-content tests that moved to NVIDIA-NeMo#3686 fail against main's current prompt, which is why they live there. - `pre-commit run --all-files`: all hooks pass, no files modified. - Schema probe against main's `NeMoGymChatCompletionCreateParamsNonStreaming`: | History | Stirrup's `to_openai_messages` | `to_provider_openai_messages` | |---|---|---| | Assistant tool call + tool result | rejected (44 errors) | validates | | Assistant reasoning, no tool calls | rejected (14 errors) | validates | | Text only | validates | validates | Adding `chat_template_kwargs` is still rejected until NVIDIA-NeMo#3682. - Build script: I ran it with a stub `apptainer` that checks every `%files` source resolves from the build directory. - The new script passes for the default, a relative and an absolute output path. - Main's script fails: `gdpval_aa_v2_python_requirements.txt` is not found from the repo root. - Container: built and verified with `verify_gdpval_sandbox.py` in NVIDIA-NeMo#3333. The image was not rebuilt for this PR, and only comment lines under `containers/` have changed since. | Arch | Interpreter | Pins | apt-closure entries missing | Verifier failures | |---|---|---|---|---| | x86_64 | 3.13.5 | 419/419 | 0 | 0 | | aarch64 | 3.13.5 | 414/419 (the 5 listed exclusions) | 3 | 0 | - Core unit tests: not run. This PR changes nothing under `nemo_gym/`, `scripts/` or `tests/`. ## Rollouts **Live GDPVal smoke (final code):** all seven PRs of the series applied together on main `1c8261080`, run with the devel-only SLURM launcher on a 4-task AA-v2 comparison smoke (one 4-GPU policy node, CPU judging): rollout 4/4 tasks saved and finished (1h04m, 0 failure rows), Office preparation 7/7 converted, comparison judging 4/4 tasks judged with 0 imputed and 0 excluded, 3 wins / 1 loss. Per-task outcomes match an earlier gdpval-devel run on the same checkpoint, tasks and references. No rate limiting occurred, so the judge retry budget was not exercised live. ## Compatibility and benchmark impact - **Sandbox image** (takes effect only after a rebuild): - Existing `.sif` images keep working. - Build from `containers/` (`cd responses_api_agents/stirrup_agent/containers && apptainer build gdpval.sif gdpval.def`) or with the recipes script. A build started from the repo root cannot find the `%files` inputs. - Python packages: - The base moves from `python:3.13-bookworm` to `debian:trixie`, with Python fixed at 3.13.5. - Python packages become exact pins. Every Python package the old definition installed is still in the pin set. - apt: `unar` is dropped (it isn't in the AA v2 closure), and three packages move to their trixie names. - Deliberate differences from AA v2: we keep a compiler toolchain, we add `espeak-ng`, and the per-command limit is 5 minutes (AA uses 10). - **GDPVal results:** runs on a rebuilt image are not comparable with runs on an image built from the old definition. Package versions, fonts and LibreOffice all change. The prompt change in NVIDIA-NeMo#3686 moves scores further. - **Token budgeting (all Stirrup runs):** - With `model_id` set and a template that renders, `max_completion_tokens` can drop below 1024 near the context limit, and a full context raises `ContextOverflowError` before the request is sent. - Without `model_id`, or when tokenization fails, the estimate is larger for every history, most for tool-heavy ones. Budgets shrink only near the context limit, and the 1024 floor still applies. - **Constructor validation:** `max_completion_tokens_cap`, `min_completion_tokens` and `min_compaction_summary_words` below 1 now raise `ValueError`. - **Logging:** a WARNING is logged every time a call uses its whole completion budget without a usable tool call, even when `truncation_recovery` is off. - **Compaction:** empty or think-only summaries now get up to 3 attempts and then fail the rollout. Before, the run continued with an empty summary. - The override uses private Stirrup internals (`_unwind_context_overflow`, `_active_tools`, `_logger`). They are stable under the existing `stirrup>=0.1.12,<0.2` pin, but a Stirrup minor release may need a follow-up. - **Provider requests:** - The non-standard message keys listed above are no longer sent, so earlier-turn reasoning is not replayed. - Unparsed tool-call blocks are removed from the provider copy of assistant text. - **Trajectories:** a repeated tool `call_id` gets a `#n` suffix from its second occurrence onward. - **`web_fetch`:** - A page with no extractable text now returns `success=False` with an error. Before, it returned an empty body. - A fragmentary extraction gets a note at the start. - **Opt-in behaviour:** `truncation_recovery` is off by default. When on, the recovery instruction is appended as a user message, so the provider can see two user messages in a row. - NVIDIA-NeMo#3686 turns it on for GDPVal and also sets `min_completion_tokens: 8192` and `min_compaction_summary_words: 50`. Other Stirrup users are unaffected. - Until NVIDIA-NeMo#3686 lands, those keys in agent YAML have no effect: `StirrupAgentWrapperConfig` accepts extra keys and ignores them. ## Checklist - [ ] I have read the [contributing guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Documentation added or updated, or N/A with justification. - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format checks pass). - [x] New source files include the required Apache-2.0 SPDX header. - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). Signed-off-by: Serge Panev <spanev@nvidia.com>

…ract (NVIDIA-NeMo#3684) ## What does this PR do? Rollout collection gets a dispatch budget and an opt-in retry contract for two kinds of sidecar and main-file rows that block a correct resume. Both came out of long GDPVal runs on fixed-walltime allocations (`gdpval-devel` NVIDIA-NeMo#2338 and NVIDIA-NeMo#2588). This is NVIDIA-NeMo#3684 of a series that moves `gdpval-devel` to main, split by file. **1. Dispatch budget (NVIDIA-NeMo#2338; validation and drain handling from NVIDIA-NeMo#2588).** Collection keeps starting rows until the queue is empty, even when the allocation is about to end. Rows started near the end are killed part-way through and their compute is lost. New `RolloutCollectionConfig` fields, all off by default: - `dispatch_budget_s` (>= 0): seconds after collection starts during which rows may start. After that, queued rows are drained: they are not started and write nothing, so the next `resume_from_cache` run starts them. The check runs after a row gets its concurrency slot, so it sees the time actually left. It requires `num_samples_in_parallel`, because unbounded dispatch would POST the whole queue before the budget is checked again. - `drain_margin_s` (>= 0): also stop starting rows when less than this much time is left. Unset, the margin is the run's own p75 rollout duration once 5 rollouts have finished. - `dispatch_longest_first`: on resume, rows whose earlier failed attempts recorded the longest `elapsed_seconds` (top level or in `response.metadata`) start first. Rows with no timing keep their input order and start after them. Stirrup on main records `elapsed_seconds` in `response.metadata` only for runs that produced a response; NVIDIA-NeMo#3686 adds it to timed-out failure rows, so without G-A timed-out tasks have no timing to sort by. - A drained row resolves to a Gym-built result: `_ng_failure_class="cancelled"` (the shared name in `nemo_gym.failure_kinds`), plus `_ng_dispatch_drained` and `_ng_no_persist`. It is written nowhere and counted as omitted. Nothing ran, so it skips model-call capture, token-capture finalization, the progress metrics and the rollouts upload. This keeps a burst of drained rows from tripping `token_id_capture.max_mask_fraction`. - `DispatchLatencyTracker` prints the per-rollout latency median, p90, p99, max, the task-hours delivered and the number of rows not started when a run ends. Rollouts/min alone hides that each rollout gets slower as concurrency rises. A run where nothing produced a result now says how many rows the budget never started. - Rows now start in input order. `asyncio.as_completed` builds a `set()` of its awaitables, which throws the order away, so `dispatch_longest_first` did nothing until the tasks were created in order. They are still created on the first iteration, as before, so `run_examples()` can be called outside a running event loop. - `run_examples()` and `_run_examples_with_metadata()` take keyword-only `dispatch_budget_s`, `drain_margin_s` and `latency_tracker`. **2. Opt-in failure retry contract (NVIDIA-NeMo#2588; made opt-in here).** Stirrup and GDPVal on main (until NVIDIA-NeMo#3686) write two kinds of rows that block a correct resume: Stirrup stamps per-attempt `timeout_exceeded` failures `_ng_failure_terminal`, and GDPVal invalid judge responses land in the main file as zero-reward rows. Other environments on main depend on today's behaviour on purpose: pinchbench stamps its timeouts terminal, and aalcr and frontierscience_judge score an invalid judge response as zero. So both fixes are behind flags that default to off, and NVIDIA-NeMo#3686 turns them on for GDPVal: - `retry_terminal_timeouts` (collection and reverification): on resume, sidecar rows of class `timeout_exceeded` are retried even when stamped terminal, up to `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`. The GDPVal environment faults `reference_missing`, `eval_missing` and `transport_ineligible` are retried the same way, and `skipped` rows never are, stamped or not. With the flag off, a row is terminal exactly when it is stamped `_ng_failure_terminal`, as on main. - `retry_invalid_judge_responses`: on resume, and before seeding in `reverify --judge-failed-only`, `migrate_invalid_judge_main_rows` moves `invalid_judge_response` rows out of the rollouts JSONL into the failures sidecar as retryable `judge_invalid` attempts. The sidecar is written and fsynced first, then the main file is atomically replaced. A marker keyed on (stage, task, rollout, attempt) makes a repeated or interrupted migration idempotent. A row whose deliverable still exists is marked `reuse_cached_deliverable`, so it is judged again without rerunning the policy. In reverification, a fresh invalid judge result goes to the sidecar as `judge_invalid`, or as terminal `permanent` if the verifier sets `invalid_judge_retryable=false`. - These class names are `_ng_failure_class` labels that predate `nemo_gym/failure_kinds.py` (NVIDIA-NeMo#3213) and are not registered there. The docstrings record the mapping: `timeout_exceeded` is `agent_timeout`, `judge_invalid` is `judge_unparseable`, the three GDPVal faults have no shared name (`gdpval:<kind>` if namespaced), and `permanent` names no kind; the row's terminal stamp is what stops the retry. As `failure_kinds` requires, retryability is decided per occurrence under an explicit opt-in, not by the name. - `is_terminal_failure`, `migrate_invalid_judge_main_rows`, `observed_elapsed`, `get_max_rollout_attempts` and `DispatchLatencyTracker` are public because the GDPVal multistage orchestrator (NVIDIA-NeMo#3687) uses them. `_get_max_rollout_attempts` stays as an alias. **3. `nemo_gym/deliverables.py` (NVIDIA-NeMo#2588).** `IGNORE_FILES` (the agent run-state files `finish_params.json`, `history.json`, `history.pkl`, `inprogress_history.json`, `metadata.json`, `log.txt` and `reference_files`) and `is_deliverable(path)`. The migration uses it to decide `reuse_cached_deliverable`, and NVIDIA-NeMo#3686's Stirrup `file_reader` and GDPVal `media` use it, so rollout collection, Stirrup and GDPVal classify files the same way. The names are the run state Stirrup writes next to its output today; they live in core because collection needs the same answer. **4. Smaller changes that apply without opt-in (NVIDIA-NeMo#2588):** see Compatibility. Docs: the "Resume interrupted runs" section and the `gym eval reverify` table in `fern/versions/latest/pages/reference/cli-commands.mdx` cover the new options, the positive `--concurrency`, the aggregate-metrics cleanup and the `--judge-failed-only` changes. Issue: none. This moves work reviewed and run on `gdpval-devel` (NVIDIA-NeMo#2338, NVIDIA-NeMo#2588). Dependencies: none. NVIDIA-NeMo#3686 depends on this PR (it imports `nemo_gym.deliverables`, and its GDPVal config turns on both retry flags). NVIDIA-NeMo#3687 depends on it (the multistage orchestrator imports the five public helpers and calls `_run_examples_with_metadata(..., dispatch_budget_s=..., drain_margin_s=..., latency_tracker=...)`). ## Validation macOS, Python 3.13, venv from `uv venv -q --python 3.13 .venv && uv sync -q --extra dev`. This PR alone on base `1c8261080`: - `pytest tests/unit_tests -q -p no:cacheprovider -n 8 -rfE`: 5700 passed, 62 failed, 161 skipped, 142 subtests passed. Every failure is a known macOS-only failure that also fails on plain main: `telemetry/test_memory.py` (2), `test_ci_environment.py` (1), the `test_cli.py` memory-profiler test (1), `test_enroot_provider.py` (1), `test_opensandbox_compose.py` (11), and `test_super_vllm_launcher.py` (4 tests plus launcher subtests, whose failing parameters vary from run to run). - `pytest tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_dispatch_latency.py tests/unit_tests/test_rollout_reverification.py tests/unit_tests/test_rollout_health.py tests/unit_tests/test_deliverables.py -q -p no:cacheprovider -n 8`: 542 passed. - Same five files with `--cov=nemo_gym`: every added executable line runs (`rollout_collection.py` 191/191, `rollout_reverification.py` 44/44, `deliverables.py` 5/5); the files are at 95.97%, 100% and 100%. `_run_examples_with_metadata` and `run_examples` are `# pragma: no cover` as on main, so the drain check and in-order scheduling are tested (through `run_examples` and `run_from_config`) but not counted. - Environments whose behaviour the default-off flags protect: `pytest responses_api_agents/pinchbench/tests` 86 passed, `pytest resources_servers/frontierscience_judge/tests` 45 passed, `pytest resources_servers/aalcr/tests/test_app.py` 25 passed. - `pre-commit run --all-files`: all hooks pass, no files modified. With the rest of the series applied (the combined snapshot plus these changes): - `pytest tests/unit_tests -q -p no:cacheprovider -n 8 -rfE`: 5712 passed, 61 failed (the same known macOS set), 161 skipped, 143 subtests passed. - `cd resources_servers/gdpval && pytest tests -q -p no:cacheprovider`: 564 passed. - `cd responses_api_agents/stirrup_agent && pytest tests -q -p no:cacheprovider`: 319 passed, 2 skipped. Tests: 60 new test functions and 1 renamed (78 new test cases once parametrized; the four test files go from 396 to 474 collected). They cover: - the budget: no budget, spent, zero, margin, generous, the concurrency validator, negative values rejected, the startup message for an unset, explicit and zero margin; - drained rows end to end through `run_from_config` with token capture on and `max_mask_fraction` set: only the 3 rows that ran are finalized, persisted and counted in progress, 4 are reported as not started, and a run where every row drained says so; - rows reach the server in list order, and `run_examples()` can be built outside a running event loop; - longest-first ordering, including known rows ahead of unseen ones and the longest failed attempt winning; - the terminal classifier with the flag off and on, and the config defaults being off; - invalid-judge migration: distinct attempts, per-stage keys, reuse only when a real deliverable exists, valid rows kept byte for byte, rows left alone by default, and an interrupted migration that leaves the main file intact and resumes without duplicating the sidecar row; - reverification: invalid results scored by default and sent to the sidecar when opted in, migration before seeding only when opted in, incremental `--judge-failed-only --resume` recovery on both the default `judge_failed` path and the opt-in path, multi-stage rows rejected before any file is written, and verifier inputs passed through; - the latency tracker's quantiles, drain margin and summary, and `is_deliverable`. ## Rollouts **Live GDPVal smoke (final code):** all seven PRs of the series applied together on main `1c8261080`, run with the devel-only SLURM launcher on a 4-task AA-v2 comparison smoke (one 4-GPU policy node, CPU judging): rollout 4/4 tasks saved and finished (1h04m, 0 failure rows), Office preparation 7/7 converted, comparison judging 4/4 tasks judged with 0 imputed and 0 excluded, 3 wins / 1 loss. Per-task outcomes match an earlier gdpval-devel run on the same checkpoint, tasks and references. No rate limiting occurred, so the judge retry budget was not exercised live. With every new option left at its default, `gym eval run` writes the same rollout rows as before; what differs is the order in which rows start and the latency summary at the end of the log. ## Compatibility and benchmark impact New options, all off or `None` by default, which keeps main's behaviour: - `RolloutCollectionConfig`: `dispatch_budget_s`, `drain_margin_s` (ignored unless `dispatch_budget_s` is set), `dispatch_longest_first`, `retry_terminal_timeouts`, `retry_invalid_judge_responses`. - `RolloutReverificationConfig`: `retry_terminal_timeouts`, `retry_invalid_judge_responses`. - `run_examples()` / `_run_examples_with_metadata()`: the new parameters are keyword-only, so existing calls still work, and the returned iterator still starts rows only when it is first consumed. Changes that apply without opting in: - **`num_samples_in_parallel` must be > 0.** `0` or a negative value now fails config validation in `gym eval run` (`--concurrency`). Before, `0` meant no limit. No config or script in the repo sets `0`. - **Dispatch order:** rows get their concurrency slot in input order. Before, the order came from a `set()` and was effectively arbitrary. The rollouts JSONL is still written in completion order, and aggregation still sorts by `(task_index, rollout_index)`. - **Fresh runs** also delete `<stem>_aggregate_metrics.json`, next to the failures sidecar that main already deletes (NVIDIA-NeMo#2723). - **Logs:** every collection run ends with the latency summary. On resume, the sidecar-terminal line reads `(for example skipped)` instead of `(timeout_exceeded / skipped)`. - **Resume:** a `reuse_cached_deliverable` flag on a sidecar row is copied onto the retried row. Only Stirrup reads this key. On main its failure rows carry it only when the original request set it; with NVIDIA-NeMo#3686, Stirrup sets it on judge-failure rows whose deliverable exists, so their retries judge the saved deliverable instead of rerunning the policy. - **Reverification payloads:** `/verify` now also receives `deliverables_dir` and `reference_ids` when the rollout row has them, which today means GDPVal rollouts. GDPVal's verify request already declares both fields. On main, `gym eval reverify` of GDPVal rollouts therefore judged the response text alone in rubric mode and compared against every configured reference in comparison mode; with this PR it judges the rollout's own files against its own reference subset, as the original run did. **Re-verified GDPVal scores change with no flag set.** Other environments' rollouts do not carry these keys. - **`reverify --judge-failed-only`** also recovers `judge_invalid` sidecar rows. With this PR alone those only exist after opting in; with NVIDIA-NeMo#3686, Stirrup writes them for GDPVal without any flag. It also refuses multi-stage rows (`stage_index`) before writing anything and points to resuming the multi-stage collection. - **Names in shared code:** with `retry_terminal_timeouts=true`, `is_terminal_failure` names GDPVal's `reference_missing`, `eval_missing` and `transport_ineligible` classes. Nothing reads them with the flag off. Benchmark results: `gym eval run` results do not change for any environment by default. GDPVal results change once NVIDIA-NeMo#3686 turns both flags on: invalid judge responses leave the score and are judged again instead of counting as zero reward, and terminal timeouts are retried up to `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`. `gym eval reverify` results change for GDPVal as described above. ## Checklist - [ ] I have read the [contributing guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Documentation added or updated, or N/A with justification. - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format checks pass). - [x] New source files include the required Apache-2.0 SPDX header. - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Serge Panev <spanev@nvidia.com>

…IA-NeMo#3686) ## What does this PR do? This is NVIDIA-NeMo#3686 of the series that moves `gdpval-devel` to main, cut by file. It covers GDPVal judging in the resources server, the GDPVal benchmark config and README, the local-judge overlays, and the Stirrup agent's GDPVal task code (`app.py`, `file_reader.py`, `prompts/`). Every file is copied whole from one snapshot: `gdpval-devel` merged with main, plus NVIDIA-NeMo#2046's review fixes and pre-promotion fixes, including the fixes listed at the end of this section. The work comes from devel PRs NVIDIA-NeMo#2103, NVIDIA-NeMo#2331, NVIDIA-NeMo#2338, NVIDIA-NeMo#2588, NVIDIA-NeMo#3086, NVIDIA-NeMo#3121, NVIDIA-NeMo#3131, NVIDIA-NeMo#3184, NVIDIA-NeMo#3335 and NVIDIA-NeMo#3648, and from NVIDIA-NeMo#2046. **Judge transport per provider (comparison mode; NVIDIA-NeMo#2588, NVIDIA-NeMo#3131)** - Each panel member (GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.8) has its own fixed-model `openai_model` proxy. All three default to the same endpoint (`JUDGE_BASE_URL`). - Each member declares how it receives PDFs and rendered Office files (`media_mode`: `native_pdf`, `images_and_text` or `native_pdf_overflow_images`), plus provider limits: native-PDF pages, documents and bytes, image bytes, video count, raster DPI tiers, and a serialized-request cap. - A deterministic preflight removes any member that can't carry a matchup within its limits. If every member is removed for every matchup, the row fails as terminal `transport_ineligible`. - These per-member settings apply to comparison mode only. Rubric mode samples from the same panel but sends every member the server-level `judge_media_mode` (native PDFs by default), with no per-member limits or preflight. - Audio and video capability are separate flags (`handles_audio`, `handles_video`, `on_missing_av_judge`). Video goes to video-capable members. Audio that the routed member can't read is dropped with a warning. - Attachments are bounded per file, per section and per request (raw and encoded bytes), with ZIP member limits and text caps. An attachment left out for size is replaced by a marker the judge sees. - Legacy `.doc`, `.ppt` and `.xls` files are preconverted. When two Office files share a stem, each gets its own `<name>.<ext>.pdf` (NVIDIA-NeMo#2331). A failed OOXML conversion is retried once from a python-docx/python-pptx/openpyxl re-save. **Validity of judge results (NVIDIA-NeMo#2588, NVIDIA-NeMo#3648)** - Rubric: a truncated, scoreless or unparseable reply is flagged `invalid_judge_response` with a `scoring_error` instead of scoring 0 or a partial score. - Structured rubric only (`rubric_scoring_mode: structured`): negative criterion points no longer count toward `max_possible` (NVIDIA-NeMo#3648). - Comparison: the verdict is the last `BOXED[...]` token, a reply with no verdict is dropped instead of counted as a tie, and the prompt ends with a final-verdict reminder. - A request that is too large or overflows the context window fails as permanent. Throttling is retried; rubric retries now also match 502, "bad gateway" and "too many requests". - `strict_comparison_trials` (opt-in) rejects a row unless every planned vote is valid. - Stirrup turns an invalid judge response into a retryable `judge_invalid` failure that keeps the finished deliverable (`reuse_cached_deliverable`), so the retry re-judges it without re-running the policy. - Deliverable conversion runs off the event loop. **Local judges (NVIDIA-NeMo#2046, NVIDIA-NeMo#2103)** - `judge_media_mode: images_and_text` rasterizes PDF and Office pages with PyMuPDF and attaches the extracted text (`media_conversion.py`). `python -m resources_servers.gdpval.inspect_conversion` writes out what an `images_and_text` judge receives. - Overlays for a Kimi-K2.6 judge that gym spawns (`gdpval_kimi_local_judge.yaml`, `local_vllm_model/configs/moonshotai/Kimi-K2.6.yaml`) and for a self-hosted MiniMax-M3 endpoint (`gdpval_minimax_selfhosted_judge.yaml`). **Inputs (NVIDIA-NeMo#3131, NVIDIA-NeMo#3335)** - `judge_reference_files_recursive` (opt-in) includes nested files under a task's `reference_files/`. `judge_reference_files_from_eval` (opt-in) reads `reference_files/` from the eval task directory instead of each reference model's directory. - `count_eval_missing_as_loss` with `missing_eval_task_ids` (opt-in, multi-stage `stage_index` 1 only) scores a listed task with no eval deliverable as a loss. The resources server and the Stirrup agent must both set them. - `setup_libreoffice.py` skips the reinstall when LibreOffice's `javaldx` works, and the apt timeout is 1800 s. **Stirrup GDPVal task code (NVIDIA-NeMo#2338, NVIDIA-NeMo#2588, NVIDIA-NeMo#3121, NVIDIA-NeMo#3131, NVIDIA-NeMo#3184)** - New agent config fields, defaulting to the old behaviour: `context_window_tokens` (passed as the client's `max_tokens`), `min_completion_tokens`, `prompt_estimator_truncate_history_thinking` and `truncation_recovery` (passed to NVIDIA-NeMo#3685's client), and `min_compaction_summary_words` (passed to NVIDIA-NeMo#3685's `NeMoAgent`). - The GDPVal prompts (`prompts/`, moved here from NVIDIA-NeMo#3685 so they land with the code that builds their reference paths) name the registered `code_exec` tool instead of `run_shell`, list only packages the AA v2 sandbox installs, and state the `/root` working directory, the 5-minute command limit and that shell state does not persist. Reference files are listed relative to the sandbox working directory, without the `gdpval_ref_files_<id>/` prefix. - The verify cache is keyed by the row's namespace. The file reader bounds text extraction, sniffs unknown extensions for text, lists archive contents, and degrades without LibreOffice. **Benchmark config** - Starts the three member proxies alongside `gdpval_judge_model`, each with `max_http_attempts: 1800`, and sets each member's media limits. - Claude judge: `thinking: {type: adaptive}` with `output_config.effort: high` and no temperature. - `judge_pdf_max_pages: 1000`. - Stirrup: `min_completion_tokens: 8192`, `context_window_tokens: 262144`, `min_compaction_summary_words: 50`, `truncation_recovery: true`. - NVIDIA-NeMo#3684's `retry_terminal_timeouts` and `retry_invalid_judge_responses`, both `true`. **Fixes made for this PR** - The removed `handles_audio_video` member key is now migrated to both new flags with a deprecation warning; combining it with either new flag is an error. Before, it was silently ignored. - `inspect_conversion.py` crashed at start (`ImportError` on `IGNORE_FILES`). It and `comparison.py` now read `IGNORE_FILES` from `nemo_gym.deliverables`. - README and config comments corrected: per-member proxies, Claude's adaptive thinking, the comparison-only scope of per-member transport, how the page cap works in comparison mode, the judge timeout and retry layers, and the Kimi overlay. The README now also documents the judge input options, attachment environment variables and aggregate metrics. - New tests for the overlays, the comparison retry loop, truncated rubric JSON, the inspector, and the Stirrup client and agent arguments. - The NVIDIA-NeMo#3666 media-preparation helpers (`media.py`, and `check_reference_inputs` and `office` moved into it from `preconvert.py`) have no caller in this series and stay on `gdpval-devel`. Issue: none. Most of this was reviewed in the devel PRs above; the fixes in the last group are new here. Related: NVIDIA-NeMo#2046, whose changes and review fixes are included. **Depends on NVIDIA-NeMo#3682, NVIDIA-NeMo#3685 and NVIDIA-NeMo#3684; merge after them.** It does not need NVIDIA-NeMo#3681 or NVIDIA-NeMo#3683. - NVIDIA-NeMo#3684: `comparison.py` and Stirrup's `file_reader.py` import `nemo_gym/deliverables.py` at module scope, so both suites fail to collect without it. The benchmark config sets C3's two retry flags, which main's config silently ignores. C3's rollout collection also carries `reuse_cached_deliverable` onto the retried row; without it a `judge_invalid` retry re-runs the policy. - NVIDIA-NeMo#3682: the Claude judge sends `thinking` and `output_config`, which main's strict chat schema rejects with a 422. Main's audio part also accepts only wav and mp3. - NVIDIA-NeMo#3685: Stirrup's `app.py` passes the new arguments to NVIDIA-NeMo#3685's client and agent; main's client raises `TypeError: ... unexpected keyword argument 'min_completion_tokens'`. The prompt tests read NVIDIA-NeMo#3685's vendored sandbox manifests. - Open on NVIDIA-NeMo#3682: under its current default (`forward_request_chat_template_kwargs: false`), `vllm_model` drops request `chat_template_kwargs`. The thinking-off half of `truncation_recovery` then does not reach a policy served by `vllm_model`; only the one-turn instruction applies. This config does not set the flag. ## Validation macOS, Python 3.13. Each server suite ran in a venv built from that server's `requirements.txt`, with `PYTHONPATH` set to the tree under test. ```bash cd resources_servers/gdpval && python -m pytest tests -q cd responses_api_agents/stirrup_agent && python -m pytest tests -q python -m pytest tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_openai_utils.py tests/unit_tests/test_deliverables.py -q ``` | Tree | GDPVal | Stirrup | |---|---|---| | main `1c8261080` + NVIDIA-NeMo#3681, C2, C3, S and this PR | 414 passed | 321 passed, 2 skipped | | Same, with `cryptography` not importable | 411 passed, 3 skipped (`test_media_conversion` text extraction) | — | | Full series, including NVIDIA-NeMo#3687 | 603 passed | 321 passed, 2 skipped | The three core unit files: 507 passed on the first tree. Each dependency was confirmed by removing one PR's files: | Removed | GDPVal | Stirrup | |---|---|---| | NVIDIA-NeMo#3681 | no change | no change | | NVIDIA-NeMo#3682 | 5 failed: 4 cases of `test_benchmark_panel_reaches_intended_upstream_models_and_credentials`, and `test_every_supported_audio_extension_passes_proxy_ingress` | no change | | NVIDIA-NeMo#3684 | collection errors (`No module named 'nemo_gym.deliverables'`) | collection errors, same cause | | NVIDIA-NeMo#3685 | no change | 5 failed: 4 prompt tests that read NVIDIA-NeMo#3685's manifests, and `test_runtime_knobs_reach_the_client_and_agent` (the `TypeError` above) | | This PR alone | collection errors, same cause | collection errors, same cause | `pre-commit run --files` on this PR's files passed with no changes; the checklist item below is scoped the same way. `uvx ruff@0.9.9 format --check` and `check` are clean on all 44 files, and new files carry the SPDX header. ## Rollouts **Live GDPVal smoke (final code):** all seven PRs of the series applied together on main `1c8261080`, run with the devel-only SLURM launcher on a 4-task AA-v2 comparison smoke (one 4-GPU policy node, CPU judging): rollout 4/4 tasks saved and finished (1h04m, 0 failure rows), Office preparation 7/7 converted, comparison judging 4/4 tasks judged with 0 imputed and 0 excluded, 3 wins / 1 loss. Per-task outcomes match an earlier gdpval-devel run on the same checkpoint, tasks and references. No rate limiting occurred, so the judge retry budget was not exercised live. This PR stays a draft until NVIDIA-NeMo#3682, NVIDIA-NeMo#3685 and NVIDIA-NeMo#3684 merge. ## Compatibility and benchmark impact GDPVal scores after this PR are not comparable with main's. Re-baseline rubric runs and comparison/ELO runs. **Configuration** - Panel members: `handles_audio_video` is replaced by `handles_audio` and `handles_video` (the old key is migrated with a warning). Member names must resolve uniquely, or verify raises `ValueError`. A `native_pdf_overflow_images` member needs `max_native_pdf_pages` and `max_native_pdf_bytes_per_document`. - New resources-server fields (defaults): `judge_media_mode` (`native_pdf`), `judge_pdf_render_dpi` (144), `judge_pdf_max_pages` (50; the benchmark sets 1000), `judge_pdf_include_text` (true), `judge_max_images_per_request` (450), `judge_handles_audio` / `judge_handles_video` (false), `on_missing_av_judge` (`warn`), `judge_reference_files_recursive` / `judge_reference_files_from_eval` (false), `strict_comparison_trials` (false), `count_eval_missing_as_loss` (false), `missing_eval_task_ids` (`[]`), and the per-member transport fields (unset means no limit). - In comparison mode a raster judge renders up to `max(judge_pdf_max_pages, judge_max_images_per_request)` pages per file, and is not used for a matchup whose pages and images exceed `judge_max_images_per_request`. The overlays' `judge_pdf_max_pages: 30` therefore bounds rubric mode only. - New environment variables: `GDPVAL_JUDGE_REQUEST_TIMEOUT_SECONDS` (comparison default goes from 120 s to 300 s; rubric is 1800 s, where the SDK's 600 s with two retries applied before), `GDPVAL_MAX_FILE_BYTES_FOR_JUDGE` (250 MiB, as before, now overridable), the five section/total attachment and request limits, `JUDGE_GPT_MODEL`, `JUDGE_GEMINI_MODEL`, `JUDGE_CLAUDE_MODEL`, `JUDGE_GEMINI_API_KEY` (falls back to `JUDGE_API_KEY`) and `GDPVAL_GEMINI_MAX_CONCURRENT_REQUESTS` (2). - Judge SDK clients use `max_retries=0`; GDPVal's own loops and each proxy's `max_http_attempts` retry instead. `max_http_attempts: 1800` gives about 15 minutes of retries at 0.5 s per request. Main's retry set also covers 404, 408 and 500, so a persistent 404 (such as an unknown model route) takes the whole window to surface. A comparison call times out after 300 s, is not retried, and time queued behind the Gemini proxy's 2-request limit counts toward it; the proxy can keep retrying an abandoned call until its window closes. The README documents how to lower the window per proxy. - The Claude proxy's `extra_body` sends `timeout: 900` and `request_timeout: 900` in every request, as gateway-side keys for the default endpoint. Endpoints that reject unknown fields need them removed. - The benchmark config starts `gdpval_gpt55_judge_model`, `gdpval_gemini31_judge_model` and `gdpval_claude48_judge_model` alongside `gdpval_judge_model`. With `retry_terminal_timeouts: true`, `reference_missing`, `eval_missing` and `transport_ineligible` are terminal within a run and re-checked on `--resume` (NVIDIA-NeMo#3684). **Changes that move GDPVal scores** - Rubric: invalid judge replies no longer score 0 or a partial score. The resources server's rubric `aggregate_metrics` leaves `invalid_judge_response` rows out of the mean for any agent, and adds `rubric/aggregate_rows_total`, `rubric/aggregate_rows_included`, `rubric/legacy_invalid_rows_excluded` and `rubric/aggregate_rows_included_fraction`. - Structured rubric: negative points are excluded from `max_possible`. Binary mode is unchanged. - Comparison verdicts: last `BOXED[...]` token, no-verdict replies dropped, final-verdict reminder in the prompt. `eval_missing` (no `finish_params.json`) was scored as a loss; it is now a terminal failure in the sidecar, like `reference_missing` and `transport_ineligible`. - Comparison payloads: an Office file's rendered PDF is attached once (main attached it under both names). Office files with no render and file types the judge can't read are left out instead of sent as filename stubs. Every rendered `.xlsx` also carries a structured-cells text block (formulas and cached values, up to 120k characters). Each member sees PDFs in its own representation and within its own limits. - Comparison aggregates: when rows carry `expected_final_stage_index` (written by NVIDIA-NeMo#3687), the headline `comparison/eval_elo` comes from that stage. If it is missing, incomplete or unfit, the headline is withheld (`comparison/final_stage_degraded=1`) rather than taken from another stage; rows without the field keep the last-stage headline. New keys: `comparison/stage_<k>/{judged_tasks,judged_votes,imputed_loss_tasks,imputed_loss_votes}`, `comparison/headline_stage_index`, `comparison/final_stage_*` and `comparison/expected_final_stage_*`. `comparison/win_rate` is omitted when no vote was judged. `judge_response` gains `total_invalid` and a per-judge `invalid_count`. - Claude judge: adaptive thinking with high effort and no temperature (was `thinking: enabled`). - Policy (set in the GDPVal config): `min_completion_tokens: 8192` (the code default stays 1024), `min_compaction_summary_words: 50` and `truncation_recovery: true`. The rewritten prompt names the real tool, package set and working directory, and its reference-file paths exist in the sandbox. **Every `stirrup_agent` config** - `timeout_exceeded` is no longer stamped terminal; on `--resume` it is retried up to `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`. - A verify result with `invalid_judge_response` becomes a failure instead of a scored row: retryable `judge_invalid`, or terminal `permanent` when the server sets `invalid_judge_retryable: false`. - A `/verify` error whose text matches GDPVal's permanent markers (413, "maximum context length", "too many tokens", "input is too long" and similar) is stamped terminal `permanent` and not retried, unless a throttle marker is also present. - The request's `max_output_tokens` now caps completion tokens; the context window comes from `context_window_tokens` (default 262144). - Failure rows after `/verify` carry the policy response plus `deliverables_dir`, `elapsed_seconds` and `_ng_timeout_inflight`. `judge_only` and `rerun_incomplete` fall back to a flat `task_<id>/` directory. - Stirrup now imports `resources_servers.gdpval` (`scoring`, `judge_panel`, `preconvert`, `media_conversion`), while the GDPVal server already imports Stirrup's file reader. **Known limitations** - `requirements.txt` lists `cryptography>=42.0`, but the root `pyproject.toml` `exclude-dependencies` drops it from server venvs, and an `overrides.txt` does not bring it back. Without it, pdfminer can't load: the `images_and_text` text copy is empty (a warning is logged), 3 `test_media_conversion` tests skip, and rubric `read_deliverable_files` returns an error string for PDFs (also true on main). The fix is a change to the root `pyproject.toml`. - Rubric judge sampling is seeded per task, so a `judge_invalid` retry draws the same member. A provider that rejects a task's payload in rubric mode fails every attempt until the attempt limit. - ZIP members skipped by the archive limits are logged but not marked in the judge request. - The README's multi-stage sections describe `partial_completion` and other behaviour that lands with NVIDIA-NeMo#3687. ## Checklist - [x] I have read the [contributing guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Documentation added or updated, or N/A with justification. - [x] Pre-commit checks pass locally (scoped: `pre-commit run --files` on this PR's files). - [x] New source files include the required Apache-2.0 SPDX header. - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). Signed-off-by: Serge Panev <spanev@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>

## What does this PR do? Makes multistage GDPVal ELO runs safe to resume across allocations, and adds the multistage MiniMax judge overlay. It carries the multistage orchestrator work from NVIDIA-NeMo#2588, NVIDIA-NeMo#3086, NVIDIA-NeMo#3131 and NVIDIA-NeMo#3615 on `gdpval-devel`, plus the overlay from NVIDIA-NeMo#2046 with the anchor refreshes from NVIDIA-NeMo#3184, NVIDIA-NeMo#3319 and NVIDIA-NeMo#3473. Multistage runs are long and get resumed across allocations. On main today: - the resume journal is keyed only on the stage shape, seed, references and task distribution, so a resume can replay a journal made from different rows, rollout settings or servers; - a stale journal deletes the rollouts, failure sidecar and journal it replaces; - every stage is marked complete whatever its failures, and the run always goes on to the final stage, so later stages are planned from an ELO fit that is missing rows; - the multistage path drops `route_failures_to_sidecar`, so a failed `/run` raises and kills the stage's in-flight rollouts. ### Changes - **Resume fingerprint.** It now also hashes: - the materialized rows, by content; - the rollout settings that affect results: `agent_name`, `limit`, `num_repeats`, `num_repeats_add_seed`, `responses_create_params`, `prompt_config`, `skills`; - every resolved server block, minus the top-level fields that change between allocations or resumes: `host`, `port`, `base_url`, `api_key`, `default_headers`, `*_base_url`, `*_api_key`, `*_default_headers`, `endpoint_*`, and the operational agent knobs `rerun_incomplete` and `concurrency`; - the `multistage.transport_assignment_repair` block, when set. Not hashed: `partial_completion`, `retry_inprocess`, `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`, and the contents of the reference deliverable directories. A new policy endpoint, rotated key, `RERUN_INCOMPLETE=true` or a different agent concurrency therefore keeps the journal. Every row carries the fingerprint as `verify_cache_namespace`, so cached `/verify` results are keyed per run. NVIDIA-NeMo#3686 adds the Stirrup code that reads it. - **Prior evidence is set aside, not deleted.** - A fresh start renames the rollouts, failure sidecar, journal and aggregate metrics to `<file>.stale.<ns>`. - Restarting from a stage moves later stages' rows to `<file>.pruned.<ns>` with atomic rewrites. The derived aggregate-metrics file is deleted. - The journal records `restart_from_stage` and `restart_cleanup_complete`, so an interrupted cleanup finishes on the next start. - A completed-stage marker is discarded while the failure sidecar still holds a retryable failure for that stage; the stage then replays its recorded plan. - **Non-final stages are strict.** A non-final stage completes only when every planned row has a usable judged battle, unless it sets `partial_completion`. - A stage with a retryable or undispatched row stays open. The run writes its rollouts and aggregate metrics, logs that it stopped and how to continue (`--resume`), and exits 0 with no `comparison/eval_elo`. NVIDIA-NeMo#3686's aggregation marks that result degraded. - When a stage is rejected on coverage with nothing left to retry (a terminal row, or one that used all `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`, default 3), the run writes its rollouts and then raises instead of emitting aggregate metrics (NVIDIA-NeMo#3615). - The final stage is never held open. A final stage with missing rows is written and aggregated; NVIDIA-NeMo#3686's aggregation reports its per-stage fit, marks it degraded and emits no headline `comparison/eval_elo`. - **Retry state carries across resumes.** Attempt counts, the longest failed-attempt duration, and the latest failure and no-persist dispositions are loaded per stage from the sidecar and the journal. The attempt cap therefore holds across allocations, `dispatch_longest_first` orders resumed rows by their longest earlier attempt, and a row whose deliverable survived a failed judging attempt is re-judged rather than re-run. - **Opt-in in-process retry.** `multistage.retry_inprocess: true` re-dispatches rows that failed with `timeout_exceeded` or `transient` in the same process, up to `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`. Other retryable classes wait for a resume. - **Opt-in partial calibration.** A non-final stage can set `partial_completion` (coverage floors, `waivable_failure_classes`, `tolerate_unresolved`) to accept the stage with bounded missing evidence. - The included and omitted rows and an evidence hash are frozen in the journal and revalidated on every resume. - A freshly planned policy-enabled stage uses a seeded, balanced assignment of tasks to references. - Setting it on the final stage is rejected. - **Opt-in transport-aware reference assignment** (`multistage.transport_assignment_repair.enabled`, new `transport_assignment.py`). At stage planning it reassigns the fewest tasks needed so that each candidate/reference attachment set fits the judge's file, raw, wire and per-section limits (defaults: the constants in `comparison.py`), keeping each reference's task count exact. It makes no provider calls. - Only tasks whose candidate deliverable already exists at planning time can move. On a fresh run nothing exists yet, so the default mode changes nothing and lists every task under `unrepairable_tasks` in the stage plan's receipt. - `reference_availability_only: true` needs no candidate deliverable: it moves tasks away from references with no completed deliverable (NVIDIA-NeMo#3131). - A task with no compatible reference stays on its original draw (`unroutable_tasks`), and if no count-preserving assignment exists the whole stage keeps its original draw (`infeasible`). Planning never aborts the run; those rows fail at dispatch as they would without repair. - The candidate root is the `persist_deliverables_dir` of the agent serving the GDPVal server. It must be set and absolute, as the Stirrup agent already requires; it may not exist yet. Every reference `deliverables_dir` must be a readable directory. Config errors fail at startup, before any prior state is set aside. - **Rollout settings forwarded into the multistage path.** `route_failures_to_sidecar` (already on main) plus NVIDIA-NeMo#3684's `dispatch_budget_s`, `drain_margin_s`, `dispatch_longest_first` and `DispatchLatencyTracker` summary. `dispatch_budget_s` is one wall-clock budget per process, counted from driver start and shared by all stages. - **The driver opts in to NVIDIA-NeMo#3684's failure contract.** On the single-pass path it sets `retry_terminal_timeouts` and `retry_invalid_judge_responses` to `true` only when the config leaves them unset; an explicit `false` is kept. The multistage bookkeeping always applies the contract, so a multistage run with either key explicitly `false` fails at startup. The bookkeeping uses the helpers NVIDIA-NeMo#3684 makes public. - **New overlay `resources_servers/gdpval/configs/gdpval_minimax_multistage_judge.yaml`.** Multistage comparison against a self-hosted MiniMax-M3 judge with 18 GDPval-AA v2 anchors. - Reference directories resolve from `$GDPVAL_REFS_ROOT`. Reference deliverables are not shipped with Gym, so the overlay can't run without your own copies. - The judge proxy sets `max_http_attempts: 1800`, because main caps non-internal client retries at 3 tries (NVIDIA-NeMo#3079, NVIDIA-NeMo#3538). - Its header documents the strict first stage and the resume command. Issue: none. This is part of the series that promotes `gdpval-devel` to main split by file. NVIDIA-NeMo#2046, still open against main, adds an earlier version of the same overlay file. ### Dependencies - **NVIDIA-NeMo#3684:** imports `DispatchLatencyTracker`, `get_max_rollout_attempts`, `is_terminal_failure`, `migrate_invalid_judge_main_rows` and `observed_elapsed` from `nemo_gym.rollout_collection`, and the `retry_terminal_timeouts` / `retry_invalid_judge_responses` config fields. - **NVIDIA-NeMo#3686:** the judge size limits in `comparison.py` that transport repair reads, the aggregation of `expected_final_stage_index`, the Stirrup reader of `verify_cache_namespace`, and `benchmarks/gdpval/config.yaml`, which the fingerprint test loads. The other PRs in the series come in through NVIDIA-NeMo#3686. - Merge order matters: on main alone, `multistage_orchestrator.py` fails to import (`DispatchLatencyTracker`), and main's `benchmarks/gdpval/config.yaml` already points `rollout_collection_driver` at it, so every `--benchmark gdpval` run would break. Keep this PR a draft until NVIDIA-NeMo#3684 and NVIDIA-NeMo#3686 merge. ## Validation macOS, Python 3.13. The suites ran on the promotion snapshot with this PR's fixes applied (base `1c8261080` plus every PR in the series; the two nested `pyproject.toml` files are left out). - `cd resources_servers/gdpval && pytest tests -q -p no:cacheprovider`: **582 passed**. - `cd responses_api_agents/stirrup_agent && pytest tests -q -p no:cacheprovider`: **320 passed, 2 skipped**. - This PR's test modules (`test_multistage_orchestrator.py`, `test_multistage_elo.py`, `test_transport_assignment.py`, `test_hardening_fixes.py`): **221 passed**. - The same tree with this PR's files reverted to base or removed (base plus the other PRs): GDPVal **437 passed**, Stirrup **320 passed, 2 skipped**. No earlier PR depends on this one. - `pytest tests/unit_tests -q -p no:cacheprovider` (no core file changes here; run for the combined tree): 5700 passed, 161 skipped, 145 subtests passed, 59 failed (20 tests and 39 subtests). All 59 are pre-existing macOS-only failures that also fail on plain main, in `telemetry/test_memory.py`, `test_ci_environment.py`, the `test_cli.py` memory-profiler test, `test_enroot_provider.py`, `test_opensandbox_compose.py` and `test_super_vllm_launcher.py`; the subtest count varies by one or two between runs because some `test_super_vllm_launcher.py` subtests are flaky on macOS. - `uvx ruff@0.9.9 format --check` and `uvx ruff@0.9.9 check` on the changed Python files, and `pre-commit run --files <changed files>`: clean, no files modified. - New tests cover: - fingerprint: the shipped `benchmarks/gdpval/config.yaml` plus `vllm_model.yaml` keep the same fingerprint when `policy_base_url`, `policy_api_key` or `RERUN_INCOMPLETE` change and a different one when the policy model changes; bare `base_url`/`api_key`/`default_headers`/`openai_default_headers` and agent `concurrency` are ignored, `agent_max_turns` is not; - transport repair: the candidate root comes from the linked agent (a resources-server `persist_deliverables_dir` or another server's agent is ignored), relative and non-directory roots are rejected, a root that does not exist yet pins every task, all-incompatible tasks keep their draw, and a capacity-infeasible stage keeps its original draw with `infeasible` set; - failure contract: an explicit `false` survives the single-pass driver, and multistage rejects it before touching any state; - a loop-level run with one `reference_missing` row under a 0.9-floor partial policy: without `tolerate_unresolved` the stage stays open (not coverage-blocked) and stage 2 is not planned; with it the stage is accepted as partial and stage 2 runs; - the stage planner's repair seam: the repaired mapping is dispatched and its receipt recorded in the plan, and a repair that drops a task or picks a reference outside the stage raises; - the retryable-stop log line. ## Rollouts **Live GDPVal smoke (final code):** all seven PRs of the series applied together on main `1c8261080`, run with the devel-only SLURM launcher on a 4-task AA-v2 comparison smoke (one 4-GPU policy node, CPU judging): rollout 4/4 tasks saved and finished (1h04m, 0 failure rows), Office preparation 7/7 converted, comparison judging 4/4 tasks judged with 0 imputed and 0 excluded, 3 wins / 1 loss. Per-task outcomes match an earlier gdpval-devel run on the same checkpoint, tasks and references. No rate limiting occurred, so the judge retry budget was not exercised live. Until then the multistage-specific paths (resume, partial completion, transport repair, the final-stage guard) are covered by the unit tests above only. ## Compatibility and benchmark impact Only runs that go through `rollout_collection_driver: resources_servers.gdpval.multistage_orchestrator:run_rollout_collection` are affected; `benchmarks/gdpval/config.yaml` sets it. Other environments are unchanged. - **Stages no longer advance past failures (GDPVal multistage only).** On main every stage is marked complete and the run always reaches the final stage. Now a non-final stage without `partial_completion` needs a usable battle for every planned row: - one retryable or undispatched row stops the run after that stage with exit status 0 and no `comparison/eval_elo`; rerun with `--resume`, or set `multistage.retry_inprocess`; - one terminal or max-attempt row makes the run raise after writing its rollouts; - under GDPVal's failure contract `reference_missing`, `eval_missing` and `transport_ineligible` are retried across resumes, so a missing reference deliverable costs up to `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` launches before the coverage rejection. `partial_completion.tolerate_unresolved`, or `transport_assignment_repair.reference_availability_only` when another reference has capacity, avoids that. - Strictness does not apply to single-stage runs or to the final stage. - **Existing multistage journals won't resume.** The fingerprint inputs changed, so the first run after upgrading starts fresh; the old state is renamed aside. After that, changing the dataset rows, `num_repeats`, the rollout settings above, or any server setting other than the ignored fields invalidates the journal. - **Verify cache is keyed per run.** Cached `/verify` results are namespaced by the fingerprint, so judgements cached by earlier runs are not reused under `RERUN_INCOMPLETE`, and any fingerprint change re-judges everything (judge cost). - **Old files stay on disk.** Fresh starts leave `*.stale.<ns>` files and stage restarts leave `*.pruned.<ns>` files next to the output; remove them by hand. - **Changing an accepted partial policy fails closed.** Removing or changing a stage's `partial_completion` (other than `tolerate_unresolved`) after that stage was accepted as partial makes resume raise; start a new output path instead. - **Failure contract on the single-pass path.** Resumed single-pass GDPVal runs retry legacy terminal timeouts and move invalid-judge rows back to the sidecar unless the config sets those keys to `false`. `benchmarks/gdpval/config.yaml` sets both to `true` in NVIDIA-NeMo#3686. Multistage runs reject an explicit `false`. - **`route_failures_to_sidecar` now reaches multistage.** Multistage runs that set it record a failed `/run` as a sidecar row instead of raising. - **`dispatch_budget_s` needs a worker limit.** On the multistage path it requires a positive `num_samples_in_parallel`, or the run raises `ValueError`. - **New row and journal fields.** Rollout rows gain `expected_final_stage_index`, `expected_stage_row_count` and `verify_cache_namespace`; retried rows carry `_ng_attempt_index`. The journal gains `attempt_dispositions`, `partial_complete` and restart records; a repaired stage plan carries a `transport_assignment_repair` receipt. - **New opt-in settings, all off by default:** - `multistage.retry_inprocess`: `false`. - `multistage.stages[i].partial_completion`: unset. When set, defaults are `min_success_fraction: 1.0`, `min_per_reference_success_fraction: 1.0`, `min_successful_rows_per_reference: 1`, `waivable_failure_classes: [timeout_exceeded]` (only `timeout_exceeded` and `transient` allowed) and `tolerate_unresolved: false`. `tolerate_unresolved: true` waives any persisted failure class, so the coverage floors are the only guard. - `multistage.transport_assignment_repair`: unset. Keys: `enabled`, `reference_availability_only: false`, `max_file_bytes`, `max_raw_bytes`, `max_wire_bytes`, `max_section_raw_bytes` (default: the judge limits in `comparison.py`) and `framing_reserve_bytes` (default `4194304`). - **Stage planning without the opt-ins is unchanged.** Task sampling and the uniform reference draw are the same, so the same seed gives the same plan unless `partial_completion` (balanced assignment) or transport repair is enabled. - **Benchmark results:** the ELO fit math is unchanged. Rows flagged `invalid_judge_response` are now left out of pooling and retried, and partial stages fit only their included rows. With NVIDIA-NeMo#3686's aggregation, the headline `comparison/eval_elo` is emitted only when the final stage has every expected row and a finite fit. For the new overlay, its 18 fixed anchors (read from the GDPval-AA v2 board on 2026-09-09, 09-11 and 09-17) set the scale; a stale anchor shifts every candidate. Judging-score changes come from NVIDIA-NeMo#3686. - **Docs:** the overlay header documents the strict first stage and resuming. `benchmarks/gdpval/README.md` (NVIDIA-NeMo#3686) documents `partial_completion`, stage strictness, `retry_inprocess`, `transport_assignment_repair`, what the resume fingerprint ignores, and the `.stale`/`.pruned` files. The other settings are described in code docstrings. ## Checklist - [ ] I have read the [contributing guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Documentation added or updated, or N/A with justification. (Overlay header and docstrings here; the README sections for these settings — stage completion, transport-aware reference assignment, resume fingerprint — ship in NVIDIA-NeMo#3686, which owns `benchmarks/gdpval/README.md`.) - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format checks pass). (Run on this PR's files.) - [x] New source files include the required Apache-2.0 SPDX header. - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Serge Panev <spanev@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>

[GDPVal] Harden rollout and judging pipelines

f25365f

Signed-off-by: Serge Panev <spanev@nvidia.com>

Kh4L force-pushed the spanev/gdpval-production-hardening branch from 8b63300 to f25365f Compare August 18, 2026 02:51

[GDPVal] Fix resume and judge recovery edge cases

39dde57

Signed-off-by: Serge Panev <spanev@nvidia.com>

Kh4L force-pushed the spanev/gdpval-production-hardening branch from 0db371d to 39dde57 Compare August 18, 2026 20:54

github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Aug 19, 2026

github-actions Bot mentioned this pull request Aug 19, 2026

[Tracker] Pull request handoff SLA #2067

Open

[GDPVal] Keep deliverable classification dependency-neutral

3ade891

Signed-off-by: Serge Panev <spanev@nvidia.com>

agronskiy reviewed Aug 20, 2026

View reviewed changes

Comment thread nemo_gym/rollout_collection.py Resolved

github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Aug 20, 2026

[GDPVal] Allow bounded partial calibration after timeouts

14125b7

Signed-off-by: Serge Panev <spanev@nvidia.com>

agronskiy mentioned this pull request Aug 24, 2026

[GDPVal] Make partial-calibration waivable failure classes configurable #2704

Merged

agronskiy and others added 3 commits August 24, 2026 11:57

agronskiy approved these changes Aug 31, 2026

View reviewed changes

Kh4L added 2 commits August 31, 2026 13:03

[GDPVal] Harden resume safety, failure classification, and transport …

96373fb

…eligibility Signed-off-by: Serge Panev <spanev@nvidia.com>

Kh4L merged commit 2ce39b1 into gdpval-devel Sep 1, 2026
15 of 16 checks passed

Kh4L deleted the spanev/gdpval-production-hardening branch September 1, 2026 00:05

Navigation Menu

Repository navigation

[GDPVal] Harden production rollout and judging paths - #2588

[GDPVal] Harden production rollout and judging paths#2588
Kh4L merged 9 commits into
gdpval-develfrom
spanev/gdpval-production-hardening

Kh4L commented Aug 18, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Aug 18, 2026

Uh oh!

Uh oh!

agronskiy commented Aug 21, 2026

Uh oh!

Kh4L commented Aug 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Kh4L commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Behavior changes

Validation

Uh oh!

copy-pr-bot Bot commented Aug 18, 2026

Uh oh!

Uh oh!

agronskiy commented Aug 21, 2026

Uh oh!

Kh4L commented Aug 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Kh4L commented Aug 18, 2026 •

edited

Loading