Repository navigation
Conversation
Signed-off-by: Serge Panev <spanev@nvidia.com>
1 of 2 tasks
marta-sd
added a commit
that referenced
this pull request
Jul 10, 2026
…1234) ## Summary Two related changes to harmonize judge-side knobs between rubric and comparison modes: 1. **Rubric default max_tokens 8192 → 65535** (`scoring.py:score_with_rubric`, `score_with_rubric_visual`). Matches the default already in place for comparison mode (`comparison.send_judge_request`) and structured rubric (`score_with_rubric_structured`). Prevents `finish_reason: length` truncations on rubrics with many criteria when callers don't override. 2. **`create_overrides` plumbing for comparison-mode judge calls** — adds the same override mechanism to `comparison.send_judge_request` and threads it through `run_trials` and `_verify_comparison`. Mirrors the rubric path where `judge_responses_create_params_overrides` already gets merged into `chat.completions.create`. Also switches `_verify_comparison` from `overrides.get` → `overrides.pop` so `model`/`api_key` are consumed and the rest pass through (same pattern `_verify_rubric` uses post #1225). After this lands, callers can override max_tokens / temperature / top_p / etc. for the comparison-mode judge from Hydra config exactly the same way they already can for rubric mode. ## Replaces #1215 Supersedes #1215 (same change, rebased onto current main). The original branch was DIRTY against main due to overlap with #1225 (raw_responses kwarg on `run_trials`), #1228 (preconvert + `clean_up_list`), and #1229 (`JUDGE_REQUEST_TIMEOUT_SECONDS`). All three are kept; this PR adds `create_overrides` alongside. ## Test plan - [x] Existing `test_app.py::test_verify_rubric_passes_create_overrides_through` continues to verify the rubric path. - [ ] Ad-hoc smoke: run a 5-task comparison-mode rollout with `++gdpval_judge_model.responses_api_models.openai_model.max_tokens=8192` to verify the override actually reaches the comparison judge call. Signed-off-by: Serge Panev <spanev@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com>
OlegSudakov
pushed a commit
to OlegSudakov/Gym
that referenced
this pull request
Jul 15, 2026
…VIDIA-NeMo#1234) ## Summary Two related changes to harmonize judge-side knobs between rubric and comparison modes: 1. **Rubric default max_tokens 8192 → 65535** (`scoring.py:score_with_rubric`, `score_with_rubric_visual`). Matches the default already in place for comparison mode (`comparison.send_judge_request`) and structured rubric (`score_with_rubric_structured`). Prevents `finish_reason: length` truncations on rubrics with many criteria when callers don't override. 2. **`create_overrides` plumbing for comparison-mode judge calls** — adds the same override mechanism to `comparison.send_judge_request` and threads it through `run_trials` and `_verify_comparison`. Mirrors the rubric path where `judge_responses_create_params_overrides` already gets merged into `chat.completions.create`. Also switches `_verify_comparison` from `overrides.get` → `overrides.pop` so `model`/`api_key` are consumed and the rest pass through (same pattern `_verify_rubric` uses post NVIDIA-NeMo#1225). After this lands, callers can override max_tokens / temperature / top_p / etc. for the comparison-mode judge from Hydra config exactly the same way they already can for rubric mode. ## Replaces NVIDIA-NeMo#1215 Supersedes NVIDIA-NeMo#1215 (same change, rebased onto current main). The original branch was DIRTY against main due to overlap with NVIDIA-NeMo#1225 (raw_responses kwarg on `run_trials`), NVIDIA-NeMo#1228 (preconvert + `clean_up_list`), and NVIDIA-NeMo#1229 (`JUDGE_REQUEST_TIMEOUT_SECONDS`). All three are kept; this PR adds `create_overrides` alongside. ## Test plan - [x] Existing `test_app.py::test_verify_rubric_passes_create_overrides_through` continues to verify the rubric path. - [ ] Ad-hoc smoke: run a 5-task comparison-mode rollout with `++gdpval_judge_model.responses_api_models.openai_model.max_tokens=8192` to verify the override actually reaches the comparison judge call. Signed-off-by: Serge Panev <spanev@nvidia.com> Co-authored-by: Marta Stepniewska-Dziubinska <marta-sd@users.noreply.github.com> Signed-off-by: Oleg Sudakov <oleg.sudakov@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related fixes to make rubric and comparison judging share one consistent override mechanism + sane defaults.
1. Bump rubric default
max_tokens8192 → 65535resources_servers/gdpval/scoring.py::score_with_rubricresources_servers/gdpval/scoring.py::score_with_rubric_visualresources_servers/gdpval/comparison.py::send_judge_request8192 was a footgun for multi-criterion rubric tasks: the judge has to emit one
score+rationaleper criterion as JSON, which routinely truncated and forced the regex-fallback path. Historically that surfaced as ~50%invalid_judge_responseon the 220-task GDPVal validation and ~44% on 140-task Mercor pre-override; bumping the cap drops it to <1%.2. Plumb
judge_responses_create_params_overridesthrough comparison modeRubric mode already merges
create_overridesover the request kwargs inscoring.py. Comparison mode read the same config dict but only consumedmodelandapi_key— every other override (max_tokens,temperature,top_p, …) was silently dropped on the way tosend_judge_request.This PR threads
create_overridesthroughrun_trials→send_judge_requestand switchesapp.py::_verify_comparisontopopmodel/api_key (mirroring rubric) so the rest of the dict reaches the judge call.After this,
judge_responses_create_params_overrides.max_tokens=N(and any other create kwarg) applies symmetrically in both modes.Compatibility
create_overridesis added withdefault=Noneon bothrun_trialsandsend_judge_request; existing callers that pass only the positional args continue to work.dict.update(overrides win) — same pattern as rubric — so any caller already passingjudge_responses_create_params_overrides.max_tokenscontinues to win over the new default.Test plan
resources_servers/gdpval/tests/test_app.pypass an explicitmax_tokens=16384and assert it lands increate_overrides; that contract is unchanged.