[yfw]

What does this PR do?

Since #3351, one failed judge comparison fails the whole cohort. A failed comparison is one with no completed answer, or a judge HTTP request that fails (an HTTP error, a connection error or a reply that is not JSON, after any retries). All of the cohort's rollouts are then masked, even when every other comparison succeeded. With 16 rollouts and a long judge answer, one comparison that exhausts max_output_tokens on both attempts discards 15 good comparisons.

This PR masks only the two rollouts of a failed comparison, and scores every other rollout from all of its comparisons. There is no new configuration key.

• Masking: both rollouts of a comparison that raises a JudgeError are masked through judge_failsafe (reward 0.0, mask_sample, failure_kind: judge_failed). Under circular, one failed comparison (5, 6) masks rollouts 5 and 6, not all 16. Under all_pairs, or with several judge passes, a rollout with any failed comparison is masked.
• Scoring: every other rollout is scored from all of its comparisons, including those against a masked rollout, which still measure it. So every scored rollout's reward averages the same number of comparisons. The group's length and style adjustments see only the scored rollouts.
• Failing the group: only when every rollout is masked, for example when the judge is down, does the whole cohort fail as a judge failure, as today. Its reason says that every rollout was masked. The server now waits for every comparison to finish or fail, within cohort_evaluation_timeout_s.
• Unchanged:
  • An error outside the judge request still fails a /verify group with HTTP 503 at once, and cancels the comparisons still running.
  • A completed but malformed answer keeps the default-score fallback.
  • cohort_evaluation_timeout_s still bounds the group. A judge_request_timeout_s expiry fails a comparison only if it comes before that deadline; both default to 1,800 seconds, so by default the group deadline fires first and the group fails.
  • /compare has no per-response mask, so a failed comparison still returns 503 there.
• Other paths: the fixed_baseline path follows the same rule: a rollout with a failed judge pass against the baseline is masked.
• Checkpoints: a member masked alone is saved and restored as masked (masked_alone in its saved state); its siblings replay their rewards.
• Late requests: a regenerated answer for a masked member gets its recorded judge failure, as a failed group's members do. So a --resume that reruns a masked row records it as masked again, instead of getting HTTP 409 and stopping the collection.
• Metrics and logs: new genrm_failed_comparison_rate_per_group and genrm_masked_rollout_rate_per_group metrics. Each failed comparison is logged when it fails, and each masked rollout is logged with its reason.
• Why both rollouts, not a score from the comparison that is left: in a group-relative trainer, that would mix rewards averaged over two comparisons with rewards from one. Under circular, the comparison that is left also shows one rollout only as response_1 and the other only as response_2, so a judge that favors one position moves their scores in opposite directions.

Consumer requirement, and merge order. A masked rollout carries a placeholder reward of 0.0. A trainer must keep it out of the group's baseline and normalization, or it shifts its siblings' advantages. In NeMo RL that is grpo.masked_reward_policy: exclude_infra (NVIDIA-NeMo/RL#4968; judge_failed is not a policy failure kind) or exclude. Under NeMo RL's default, include, the placeholder counts. NeMo RL also needs env.should_mask_flagged_samples on, its default; with it off, NeMo RL trains the masked rows on their 0.0 rewards. So this PR should merge after NVIDIA-NeMo/RL#4968, and GenRM recipes should set exclude_infra when they move to a Gym pin that has it.

Validation

Against the final diff (rebased on 5911d628c), Python 3.13.14:

gym env test +entrypoint=resources_servers/genrm_compare +should_validate_data=true   # 276 passed
pre-commit run --all-files                                                          # passed

New tests in tests/test_failed_comparisons.py, tests/test_checkpoint.py and tests/test_http_lifecycle.py:

• Masking, 16-rollout circular group:
  • One failed comparison: its two rollouts are masked, and the other 14 are scored with their own scores.
  • The neighbors of a masked rollout are still scored from both of their comparisons; the judge stub gives each position a different score, so a lost comparison would change their rewards.
  • Two non-adjacent failures: their four rollouts are masked.
• Group of 4:
  • Two adjacent failures mask three rollouts. An exact retry replays the same outcomes without judging again.
  • The masked rollouts do not change the length adjustments of the scored rollouts.
  • all_pairs: one failed pair masks its two rollouts, and the other two are scored from all three comparisons; when every rollout is masked, the group fails.
  • Every comparison fails: the group fails.
• Errors: a non-judge error gives 503 for every member and cancels the comparisons still running.
• Logging: a failed comparison is logged when it fails, before a later non-judge error or the group deadline ends the group.
• Late requests: a regenerated answer for a masked member gets its judge failure; a scored member still refuses a different answer with 409.
• /compare: a failed comparison returns 503.
• Fixed baseline: a failed judge pass masks its rollout, also through /verify.
• Checkpoint: members masked alone and their scored siblings are exported and restored without judging again.
• HTTP through /run and the failsafe, 16 rollouts: one failed comparison, and two adjacent failures, mask exactly their rollouts with judge_failed; the other rollouts return their rewards.
• Regression check: with main's server code, 16 of these 20 test cases fail. The other 4 cover behaviour that does not change.

Rollouts

This PR's arm ran at 6aeff06e4. The next commit, 9ee859305, changes only the reply to a regenerated answer for a masked member, the reason text when every rollout is masked, docs and tests; scoring and masking are the same. The merge of main after it changes only import paths. The main arm ran at 5911d628c (the base of this PR) in an earlier run, with the same judge, policy, data and settings.

Setup

• Judge: nvidia/Qwen3-Nemotron-235B-A22B-GenRM, served by vLLM 0.27.1 with tensor parallel size 4 on one GB200 node.
• Policy: Qwen/Qwen3.5-4B with thinking off, served by vLLM on one GPU of a second node.
• Data: the 4 tasks in resources_servers/genrm_compare/data/example.jsonl. Each run holds one task with a new _ng_group_id, so it makes one 16-rollout circular group.
• Settings: the shipped genrm_compare settings, with two changes that cause real judge failures:
  • genrm_responses_create_params.max_output_tokens: 1341. We first ran 16 judge comparisons with a 16,384-token budget. The answers were 531 to 2,981 tokens long, and 1,341 is their 55th percentile. So some answers run out of budget before the verdict.
  • genrm_parse_retries: 1, so each comparison gets 2 attempts.

Commands. Each arm starts its servers once, and they stay up for all 4 tasks.

gym env start "+config_paths=[genrm_5330_run.yaml]" \
    +policy_base_url=http://<policy-host>:8001/v1 +policy_api_key=dummy +policy_model_name=policy \
    +genrm_base_url=http://<judge-host>:8000/v1 +genrm_model_name=genrm +genrm_max_output_tokens=1341 \
    ++genrm_compare_resources_server.resources_servers.genrm_compare.genrm_parse_retries=1

# Once per task, with a new _ng_group_id in task<k>.jsonl:
gym eval run --no-serve --agent genrm_simple_agent \
    --input task<k>.jsonl --output task<k>-results.jsonl \
    --num-repeats 16 --concurrency 16 +route_failures_to_sidecar=true

genrm_5330_run.yaml

genrm_compare_resources_server and genrm_simple_agent are the same as in configs/genrm_compare.yaml, with max_output_tokens: ${genrm_max_output_tokens}. The two model servers point at the external vLLM servers:

policy_model:
  responses_api_models:
    vllm_model:
      entrypoint: app.py
      base_url: ${policy_base_url}
      api_key: ${policy_api_key}
      model: ${policy_model_name}
      return_token_id_information: false
      uses_reasoning_parser: false
      chat_template_kwargs:
        enable_thinking: false

genrm_model:
  responses_api_models:
    genrm_model:
      entrypoint: app.py
      base_url: ${genrm_base_url}
      api_key: dummy_key
      model: ${genrm_model_name}
      return_token_id_information: false
      uses_reasoning_parser: false
      supports_principle_role: true
      hf_home: null
      vllm_serve_env_vars: {}
      vllm_serve_kwargs:
        tensor_parallel_size: 4
      debug: false

Results (one 16-rollout group per task and arm)

Task	This PR (6aeff06e4)	main (5911d628c)
1	4 of 16 comparisons failed: (8, 9), (9, 10), (10, 11) and (14, 15). Their rollouts, 8 to 11, 14 and 15, were masked. The other 10 were scored (3.0 to 4.5). Rates: 0.25 and 0.375.	The first failed comparison failed the group. All 16 rollouts were masked.
2	No comparison failed. All 16 were scored (3.0 to 5.25). Rates: 0.0 and 0.0.	No comparison failed. All 16 were scored (2.0 to 5.0).
3	4 of 16 comparisons failed: (0, 1), (10, 11), (12, 13) and (14, 15). Their rollouts, 0, 1 and 10 to 15, were masked. The other 8 were scored (3.5 to 5.25). Rates: 0.25 and 0.5.	The first failed comparison failed the group. All 16 were masked.
4	1 of 16 comparisons failed: (15, 0). Rollouts 15 and 0 were masked. The other 14 were scored (2.0 to 5.0). Rates: 0.0625 and 0.125.	The first failed comparison failed the group. All 16 were masked.

"Rates" are genrm_failed_comparison_rate_per_group and genrm_masked_rollout_rate_per_group, as reported on each scored row.

This PR scored 48 of the 64 rollouts, and main scored 16. On this PR, the masked rollouts are exactly the rollouts of the failed comparisons, and no group failed as a whole. On both arms, every masked row has mask_sample: true, failure_kind: judge_failed and reward 0.0, and lands in the collector's _failures.jsonl sidecar.

The two arms ran separately, and each sampled its own policy and judge answers. Tasks 1, 3 and 4 had failed comparisons on both arms, but not the same ones. On main, a group fails at its first failed comparison, so its log shows one failed comparison per masked group.

Server log lines

This PR, task 4 (one failed comparison, two rollouts masked):

GenRM output budget exhausted for pair (15, 0) (attempt 2/2, max_output_tokens=1341): response is incomplete (reason=max_output_tokens) and contains no verdict
GenRM dropped comparison (15, 0) (judge 0) of 16 after a judge failure: Judge returned no completed answer after 2 attempts (2 of them exhausted max_output_tokens=1341 without emitting a verdict; raise the budget or constrain the judge's reasoning)
GenRM masked one rollout of key='group_id::996c6e1f-f9ce-4ceb-86d6-d05f765bcdb0::group_attempt::0': GenRM rollout_index=15 lost a comparison to a judge failure (1 of 16 comparisons failed): Judge returned no completed answer after 2 attempts (...)
GenRM masked one rollout of key='group_id::996c6e1f-f9ce-4ceb-86d6-d05f765bcdb0::group_attempt::0': GenRM rollout_index=0 lost a comparison to a judge failure (1 of 16 comparisons failed): Judge returned no completed answer after 2 attempts (...)

main, task 1 (one failed comparison, group failed):

GenRM output budget exhausted for pair (7, 8) (attempt 2/2, max_output_tokens=1341): response is incomplete (reason=max_output_tokens) and contains no verdict
GenRM cohort disposition=failed key='group_id::d1c0ec60-46a2-4bd0-ba08-4757723b0524::group_attempt::0' attempt=0 members=16 kind=judge reason="Judge returned no completed answer after 2 attempts (2 of them exhausted max_output_tokens=1341 without emitting a verdict; raise the budget or constrain the judge's reasoning)"

Compatibility and benchmark impact

• Default behavior changes. A group with a failed comparison is now scored, except the failed comparison's two rollouts, instead of being fully masked.
• Trainer setting needed. The masked rollouts' placeholder reward must stay out of the group statistics (NeMo RL exclude_infra from feat(grpo): add exclude_infra masked-reward policy for infra-masked rollouts RL#4968, or exclude). Merge after feat(grpo): add exclude_infra masked-reward policy for infra-masked rollouts RL#4968.
• /compare. Unchanged: a failed comparison returns 503.
• New fields. Two per-group metrics on /verify, and masked_alone in a checkpointed member's state.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally.
• Documentation added or updated (resources_servers/genrm_compare/README.md).
• Pre-commit checks pass locally (pre-commit run --all-files).
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off (git commit -s).

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yfw]

/ok to test f6d207d

[yfw]

/ok to test a66283d

[yfw]

/ok to test d94066c

[yfw]

/ok to test 6aeff06

[yfw]

/ok to test 9ee8593

[yfw]

/ok to test 1f4ccf8

[yuhezhang-ai]

Thanks for working on this, @yfw! I left one inline comment about returning promptly after a judge failure in /compare.

I also have an earlier performance PR, #3625, that touches some of the same scoring and checkpoint code. It already incorporates main’s newer failure reporting, so I’d recommend landing that first, then rebasing this PR and rechecking the combined changes. I think that would make the overlap easier to reconcile.

Would that order work for you, or would you prefer a different approach? Happy to coordinate. We’d still keep NVIDIA-NeMo/RL#4968 ahead of this PR, as you’ve documented.

[yuhezhang-ai]

[P2] Preserve fail-fast behavior for /compare

Could we keep prompt failure/cancellation for the batch /compare path while retaining partial collection for /verify? This loop now records JudgeError and waits for the remaining comparisons, but _run_compare rejects the entire batch if any comparison failed. One immediate failure plus a slow judge call therefore delays the 503 until that call finishes; previously the remaining calls were cancelled promptly. /compare also has no enclosing cohort timeout. Please preserve the batch behavior and add a regression test with one failing and one pending judge call.