Repository navigation
[GDPVal] Align rubric judge max_tokens default with comparison mode - #1234
Merged
Merged
Conversation
Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com> # Conflicts: # resources_servers/gdpval/app.py # resources_servers/gdpval/comparison.py # resources_servers/gdpval/scoring.py
Contributor
|
/ok to test 3024e01 |
@marta-sd, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
Contributor
|
/ok to test 5ee8986 |
marta-sd
approved these changes
Jul 10, 2026
marta-sd
enabled auto-merge (squash)
July 10, 2026 07:31
OlegSudakov
pushed a commit
to OlegSudakov/Gym
that referenced
this pull request
Jul 15, 2026
…VIDIA-NeMo#1234) ## Summary Two related changes to harmonize judge-side knobs between rubric and comparison modes: 1. **Rubric default max_tokens 8192 → 65535** (`scoring.py:score_with_rubric`, `score_with_rubric_visual`). Matches the default already in place for comparison mode (`comparison.send_judge_request`) and structured rubric (`score_with_rubric_structured`). Prevents `finish_reason: length` truncations on rubrics with many criteria when callers don't override. 2. **`create_overrides` plumbing for comparison-mode judge calls** — adds the same override mechanism to `comparison.send_judge_request` and threads it through `run_trials` and `_verify_comparison`. Mirrors the rubric path where `judge_responses_create_params_overrides` already gets merged into `chat.completions.create`. Also switches `_verify_comparison` from `overrides.get` → `overrides.pop` so `model`/`api_key` are consumed and the rest pass through (same pattern `_verify_rubric` uses post NVIDIA-NeMo#1225). After this lands, callers can override max_tokens / temperature / top_p / etc. for the comparison-mode judge from Hydra config exactly the same way they already can for rubric mode. ## Replaces NVIDIA-NeMo#1215 Supersedes NVIDIA-NeMo#1215 (same change, rebased onto current main). The original branch was DIRTY against main due to overlap with NVIDIA-NeMo#1225 (raw_responses kwarg on `run_trials`), NVIDIA-NeMo#1228 (preconvert + `clean_up_list`), and NVIDIA-NeMo#1229 (`JUDGE_REQUEST_TIMEOUT_SECONDS`). All three are kept; this PR adds `create_overrides` alongside. ## Test plan - [x] Existing `test_app.py::test_verify_rubric_passes_create_overrides_through` continues to verify the rubric path. - [ ] Ad-hoc smoke: run a 5-task comparison-mode rollout with `++gdpval_judge_model.responses_api_models.openai_model.max_tokens=8192` to verify the override actually reaches the comparison judge call. Signed-off-by: Serge Panev <spanev@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com> Signed-off-by: Oleg Sudakov <oleg.sudakov@outlook.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related changes to harmonize judge-side knobs between rubric and comparison modes:
Rubric default max_tokens 8192 → 65535 (
scoring.py:score_with_rubric,score_with_rubric_visual). Matches the default already in place for comparison mode (comparison.send_judge_request) and structured rubric (score_with_rubric_structured). Preventsfinish_reason: lengthtruncations on rubrics with many criteria when callers don't override.create_overridesplumbing for comparison-mode judge calls — adds the same override mechanism tocomparison.send_judge_requestand threads it throughrun_trialsand_verify_comparison. Mirrors the rubric path wherejudge_responses_create_params_overridesalready gets merged intochat.completions.create. Also switches_verify_comparisonfromoverrides.get→overrides.popsomodel/api_keyare consumed and the rest pass through (same pattern_verify_rubricuses post [GDPVal] Add opt-in persistence of raw judge responses #1225).After this lands, callers can override max_tokens / temperature / top_p / etc. for the comparison-mode judge from Hydra config exactly the same way they already can for rubric mode.
Replaces #1215
Supersedes #1215 (same change, rebased onto current main). The original branch was DIRTY against main due to overlap with #1225 (raw_responses kwarg on
run_trials), #1228 (preconvert +clean_up_list), and #1229 (JUDGE_REQUEST_TIMEOUT_SECONDS). All three are kept; this PR addscreate_overridesalongside.Test plan
test_app.py::test_verify_rubric_passes_create_overrides_throughcontinues to verify the rubric path.++gdpval_judge_model.responses_api_models.openai_model.max_tokens=8192to verify the override actually reaches the comparison judge call.