Repository navigation
Add imo_answerbench benchmark - #1155
Merged
gwarmstrong merged 5 commits intoApr 29, 2026
Merged
gwarmstrong merged 5 commits into
gwarmstrong merged 5 commits into
Conversation
Merged
gwarmstrong
force-pushed
the
georgea/migrate-gym-imo-answerbench
branch
2 times, most recently
from
April 28, 2026 19:17
809ce81 to
fc0564d
Compare
Subclasses math_with_judge with three overrides to support Skills-style
autograder LLM judges (the "is this answer Correct or Incorrect?"
pattern, vs the Arena-Hard "are these answers equivalent?" pattern):
1. Judge prompt is loaded from prompts/judge.yaml (Skills-style
{problem} / {predicted_answer} / {expected_answer} placeholders).
A `system` key is optional; when omitted, the judge sees only a
single user turn (matches Skills' autograder pipeline).
2. On a math-verify miss, the judge sees the raw \\boxed{...} LaTeX
the model wrote, not math-verify's normalized form. math-verify
silently mangles non-numeric answers (sets, function definitions,
conditions) into a degenerate fragment which the judge then
correctly grades as wrong.
3. The autograder is unidirectional — model answer = "Model Solution",
expected answer = "Golden Answer" — so only one judge call is made
instead of math_with_judge's bidirectional swap.
Verdict tokens default to \\boxed{Correct} / \\boxed{Incorrect}, matching
the Skills autograder prompt. Subclasses can override the prompt path
and verdict labels for other autograder formats.
The bundled prompts/judge.yaml is the IMO AnswerBench autograder
(byte-identical to nemo_skills/prompt/config/judge/imo_answerbench.yaml);
swap it out by subclassing or by extending the config.
math_with_judge itself is untouched — zero impact on existing math_with_judge
consumers (aime24, aime25, dapo17k, ...).
Tests: 24 passing, 96% coverage on app.py.
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Math benchmark from google-deepmind/superhuman (`answerbench_v2.csv`,
400 short-answer IMO-style problems). Ported from NeMo Skills'
`imo-answerbench`, on top of the new `math_with_autograder` resource
server (math-verify symbolic check first, Skills-style autograder LLM
judge as the fallback).
* `prepare.py` downloads the CSV from the same pinned superhuman
commit Skills reads, so the prepared JSONL row count + content
is byte-identical to Skills' baseline.
* `prompts/default.yaml` matches Skills' `generic/math` prompt body.
Uses `{question}` instead of Skills' `{problem}` placeholder
because Gym JSONL row keys are `question`/`expected_answer`.
* `judge_gptoss20b.yaml` ships gpt-oss-20b via NVIDIA's public NIM
API as the default judge endpoint (overridable in-recipe).
* `config.yaml` references `math_with_autograder` via _inherit_from
so this benchmark cleanly composes with the shared server config
without touching it.
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
gwarmstrong
force-pushed
the
georgea/migrate-gym-imo-answerbench
branch
from
April 28, 2026 19:51
fc0564d to
1b41ea7
Compare
Without this block, ng_run launches app.py via uv run, the script defines its classes and exits cleanly with code 0 — the head server sees the subprocess exit before uvicorn boots and aborts the run with 'Process math_with_autograder finished unexpectedly!' (no Python traceback because nothing actually went wrong on the server side; it just had nothing to do). Mirrors the same one-line tail in math_with_judge/app.py. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
…headers) * Move judge_gptoss20b.yaml from benchmarks/imo_answerbench/ into resources_servers/math_with_autograder/configs/ so the server's README quickstart is self-sufficient — without this, ng_run with the server's config_paths fails at validation (`judge_model` server is referenced by math_with_autograder.yaml but only defined inside the benchmark dir). Keeps the layout consistent with omniscience. * Update math_with_autograder/README.md to chain in the bundled judge config and document the override-with-your-own-judge pattern. * Add the missing `+prompt_config=...` flag to the benchmark README's ng_collect_rollouts block. Without it the prepared JSONL rows (which only have `question` / `expected_answer` fields) fail with `KeyError: 'responses_create_params'` because the prompt template is applied at rollout time. * Strip rationale / Skills-vs-Gym comment headers from benchmarks/imo_answerbench/prompts/default.yaml and resources_servers/math_with_autograder/prompts/judge.yaml. Context belongs in the README; in-prompt narrative rots and duplicates info. * Update benchmarks/imo_answerbench/config.yaml's config_paths to point at the relocated judge config. Verified end-to-end: both READMEs now run cleanly through ng_run + ng_collect_rollouts (5-rollout smoke against integrate.api.nvidia.com). Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
bxyu-nvidia
approved these changes
Apr 29, 2026
gwarmstrong
enabled auto-merge (squash)
April 29, 2026 17:43
cmunley1
pushed a commit
that referenced
this pull request
May 1, 2026
# Add imo_answerbench benchmark and math_with_autograder resource server
Migrates the `imo-answerbench` benchmark from NeMo Skills into Gym. Adds
a new `math_with_autograder` resource server that subclasses
`math_with_judge` to plug in a Skills-style unidirectional autograder
judge ("is this answer Correct or Incorrect?" rather than the Arena-Hard
"are these equivalent?" pattern). `math_with_judge` is unchanged.
## Includes
- `resources_servers/math_with_autograder/` — new resource server,
subclass of `math_with_judge`
- Raw `\boxed{...}` LaTeX is forwarded to the judge instead of
math-verify's normalized form, so non-numeric answers (sets, function
definitions, conditions) reach the judge intact
- Single judge call (no bidirectional swap) — matches the Skills
autograder template's fixed Model / Golden role assignment
- Default judge endpoint `openai/gpt-oss-20b` via NVIDIA's public NIM
API in `configs/judge_gptoss20b.yaml` (overridable per-benchmark)
- `benchmarks/imo_answerbench/` — benchmark config, `prepare.py`, prompt
template
- Data source: `answerbench_v2.csv` from `google-deepmind/superhuman`
(pinned to the same commit Skills reads), 400 short-answer IMO-style
problems
## Validated against NeMo Skills
`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` on an 8-GPU node, 4
rollouts/task at temperature 1.0 / top_p 0.95 / top_k 20 / max output
tokens 65536. Judge model: `gemini-2.5-pro` (same judge on both Skills
and Gym sides for the parity comparison).
```
============================================================================
imo_answerbench | gemini-2.5-pro judge | 4 rollouts/task | T=1.0 top_p=0.95
| nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
============================================================================
Metric Skills Gym Delta
---------------------------------------------------------------------
pass@1[avg-of-4] / judge_accuracy 66.75% 68.69% +1.94pp
pass@4 / judge_accuracy 81.75% 82.25% +0.50pp
pass@1[avg-of-4] / symbolic_accuracy 55.69% 59.44% +3.75pp
pass@4 / symbolic_accuracy 69.25% 72.75% +3.50pp
majority@4 / symbolic_accuracy 62.81% 63.58% +0.77pp
```
The residual ~1.94 pp on `pass@1[avg-of-4]/judge_accuracy` is
structural: Gym only invokes the judge when math-verify fails, so it
never gets the chance to demote a symbolically-correct answer the judge
would flag as wrong. Skills, by contrast, runs the judge on a wider set
of rollouts. Per-task analysis confirms this — on the 100 tasks (25%)
where Skills and Gym sampled the same multisets of extracted answers
across 4 rollouts each, the verdict multisets matched exactly (0
divergence). All residual delta lives in the 300 tasks where the model
sampled different answers between the two pipelines.
A second 180-rollout verification was run to verify the refactored
version of this PR with limited rollouts (45 problems × 4 repeats):
```
============================================================================
imo_answerbench | 180 rollouts (45 problems × 4) | gemini-2.5-pro judge
============================================================================
Metric Skills Gym Delta
---------------------------------------------------------------------
pass@1[avg-of-4] / judge_accuracy 68.89% 73.33% +4.44pp
pass@4 / judge_accuracy 84.44% 86.67% +2.22pp
majority@4 / judge_accuracy 75.56% 75.56% +0.00pp
pass@1[avg-of-4] / symbolic_accuracy 53.89% 52.22% -1.67pp
pass@4 / symbolic_accuracy 66.67% 66.67% +0.00pp
majority@4 / symbolic_accuracy 60.00% 60.00% +0.00pp
```
This is subject to the same judging differences as the original run.
---------
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Co-authored-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: cmunley1 <cmunley@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add imo_answerbench benchmark and math_with_autograder resource server
Migrates the
imo-answerbenchbenchmark from NeMo Skills into Gym. Adds a newmath_with_autograderresource server that subclassesmath_with_judgeto plug in a Skills-style unidirectional autograder judge ("is this answer Correct or Incorrect?" rather than the Arena-Hard "are these equivalent?" pattern).math_with_judgeis unchanged.Includes
resources_servers/math_with_autograder/— new resource server, subclass ofmath_with_judge\boxed{...}LaTeX is forwarded to the judge instead of math-verify's normalized form, so non-numeric answers (sets, function definitions, conditions) reach the judge intactopenai/gpt-oss-20bvia NVIDIA's public NIM API inconfigs/judge_gptoss20b.yaml(overridable per-benchmark)benchmarks/imo_answerbench/— benchmark config,prepare.py, prompt templateanswerbench_v2.csvfromgoogle-deepmind/superhuman(pinned to the same commit Skills reads), 400 short-answer IMO-style problemsValidated against NeMo Skills
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16on an 8-GPU node, 4 rollouts/task at temperature 1.0 / top_p 0.95 / top_k 20 / max output tokens 65536. Judge model:gemini-2.5-pro(same judge on both Skills and Gym sides for the parity comparison).The residual ~1.94 pp on
pass@1[avg-of-4]/judge_accuracyis structural: Gym only invokes the judge when math-verify fails, so it never gets the chance to demote a symbolically-correct answer the judge would flag as wrong. Skills, by contrast, runs the judge on a wider set of rollouts. Per-task analysis confirms this — on the 100 tasks (25%) where Skills and Gym sampled the same multisets of extracted answers across 4 rollouts each, the verdict multisets matched exactly (0 divergence). All residual delta lives in the 300 tasks where the model sampled different answers between the two pipelines.A second 180-rollout verification was run to verify the refactored version of this PR with limited rollouts (45 problems × 4 repeats):
This is subject to the same judging differences as the original run.