[balajinvda]

What does this PR do?

Names GenRM output-budget exhaustion in the judge failure instead of reporting it as a generic "no completed answer", in resources_servers/genrm_compare. No behaviour change: same retries, same JudgeError, only the message and a per-attempt warning.

Why

When the GenRM model spends its whole max_output_tokens on reasoning, the model server returns HTTP 200 with a complete envelope and no verdict: status: "incomplete", incomplete_details.reason: "max_output_tokens". Since #2814 and #3351 the retry path already handles this correctly (retry, then JudgeError), but the error reads the same as an empty or garbled answer, so an operator cannot tell a budget problem from a transport or parsing one without opening the raw response.

Measured against a hosted GenRM endpoint with the candidates delivered as response_1/response_2 roles, temperature 0.6, same input each time:

max_output_tokens	runs	verdict (status: completed)	exhausted (status: incomplete, reason: max_output_tokens, no verdict)
16,384	5	2	3
24,576	4	2	2

The outcome is bimodal: the same input either converges at roughly 8 to 10k output tokens or runs away and consumes whatever budget it is given (4 of 4 still exhausted at 49,152), so the fix belongs with the judge's prompt or reasoning budget, and the message now points there.

How

• _output_budget_exhausted(raw_response): true when the Responses object is incomplete with reason max_output_tokens.
• Each such attempt logs GenRM output budget exhausted for pair (i, j) (attempt a/n, max_output_tokens=N).
• When all attempts fail without a completed answer, the existing JudgeError message gains (k of them exhausted max_output_tokens=N without emitting a verdict; raise the budget or constrain the judge's reasoning). Completed-but-empty answers keep the unchanged generic message.
• Tests: detection helper, all attempts exhausted -> message names the budget, exhausted then verdict -> normal parse, empty completed answer -> generic message. resources_servers/genrm_compare/tests: 35 passed.

Per review, the earlier opt-in budget escalation (genrm_budget_exhausted_retry_multiplier, genrm_max_output_tokens_cap) is dropped.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally (resources_servers/genrm_compare/tests, 35 passed).
• Pre-commit checks pass locally (pre-commit run --files ...: all Passed).
• All commits have DCO sign-off (git commit -s).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

Since #2814/#3351, an incomplete response is no longer silently defaulted: it's retried and then raises JudgeError("no completed answer"). Should we trim this to just naming the cause (max_output_tokens exhausted) in that JudgeError message, and drop the multiplier and cap?

[balajinvda]

Agreed, and done. Rebased onto main and trimmed to exactly that: the existing retry-then-JudgeError path is unchanged, but when the attempts came back incomplete with reason=max_output_tokens the error now says so (k of them exhausted max_output_tokens=N without emitting a verdict), and each such attempt logs a warning. The multiplier and cap are dropped. Tests updated accordingly (35 passed), pre-commit clean.

[yaoyu-33]

LGTM. The escalation knobs are removed, and this now only clarifies output-budget exhaustion while preserving the existing retry and JudgeError behavior.

[ananthsub]

/ok to test daf175a

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/37048033494