[qiaochuz-nv]

What does this PR do?

imo-bench-algebra-036 is no longer column-shifted. Its gold answer was the category label Algebra, the real answer $Y(x)=A+\frac{B}{x}-x$ was appended to the problem text the policy sees, and source was null. A correct answer therefore scored 0, and the prompt leaked the answer. The row shipped in both the math_with_autograder README sample (data/example.jsonl row 2) and the 400-row imo_answerbench benchmark that prepare.py builds.

Fixes NVBug 6900619.
Closes #4506.

Cause

The pinned upstream file (google-deepmind/superhuman@326a740a, imobench/answerbench_v2.csv) never closes the quote around this row's Problem field: lines 150–154 end with \], a newline, then ,"$Y(x)=A+\frac{B}{x}-x$",Algebra,Functional Equation,Iran 2002. Upstream tracks it as google-deepmind/superhuman#13, still open; the file's last upstream change (2026-02-13) predates the report, so there is no fixed revision to move the pin to. csv.DictReader ends the Problem at the answer's opening quote, appends \n,<answer>" to it, and shifts every later cell one column left, leaving Source empty.

benchmarks/imo_answerbench/prepare.py (added in #1155) copied each parsed row verbatim, and example.jsonl row 2 is that output byte for byte. Nothing checked the row shape; task_data.py even described source as "null in some rows", which only this broken row was. Only this row is affected: a full parse of the pinned CSV finds one misaligned row out of 400 (99 Algebra + 1 "Functional Equation" before, 100 per category after). NeMo Skills' imo-answerbench prepare reads the same CSV the same way, so it has the same row.

Fix

• prepare.py: _realign_unterminated_problem() recognises the shape (Source missing, Problem ending in \n,<answer>") and moves the cells back. The Problem keeps its trailing newline, as its neighbours do. Every row is then checked before anything is written; a row that still has a missing or extra cell raises ValueError with its problem ID instead of shipping a shifted row. The docstring records the one-row divergence from Skills.
• example.jsonl row 2 is regenerated from the realigned upstream row: question (top level and in responses_create_params.input), gold, category, subcategory, source. The other four rows are byte-identical.
• example_metrics.json is regenerated with gym dataset collate ... +mode=example_validation (category unique count 5→4; source now 5 rows / 4 unique). The source entry moves up in the file because the dataset config gained its own source field (feat(config): unify dataset source via discriminated source: block (FEP-1025) #1637) after this file was written; the writer places the column statistic in that slot.
• task_data.py: the source description drops "null in some rows".

Realigning by shape rather than hard-coding the corrected row keeps the fix tied to the upstream bytes; if superhuman#13 is fixed and the pin moves, the row parses normally and the realignment does nothing. Dropping the row would change the benchmark size, and leaving it would keep a guaranteed false zero.

Test: tests/unit_tests/test_imo_answerbench_prepare.py feeds prepare() a verbatim excerpt of the pinned CSV (header and lines 148–158: algebra-035, -036, -037) through a stubbed urlopen. It checks that algebra-036 comes out realigned and its neighbours unchanged, that another misaligned row raises before the output file is written, and that every shipped example.jsonl row has a real category, a non-empty source, a non-category gold, and that row 2 equals the realigned output.

Not changed: example_rollouts.jsonl. Its five committed rows (from #1155) have empty model output and reward 0, so they carry no grading evidence; regenerating them needs a live model and is left out of this data fix.

Before the fix

Gym 7ada5311b (main):

python -c 'import json; r=[json.loads(l) for l in open("resources_servers/math_with_autograder/data/example.jsonl")][1]; print({k: r[k] for k in ("problem_id","expected_answer","category","subcategory","source")}); print(repr(r["question"][-40:]))'

{'problem_id': 'imo-bench-algebra-036', 'expected_answer': 'Algebra', 'category': 'Functional Equation', 'subcategory': 'Iran 2002', 'source': None}
'+\\frac{b}{a}\n\\]\n,$Y(x)=A+\\frac{B}{x}-x$"'

pytest tests/unit_tests/test_imo_answerbench_prepare.py

E       AssertionError: assert 'Algebra' == '$Y(x)=A+\\frac{B}{x}-x$'
E       Failed: DID NOT RAISE <class 'ValueError'>
E           AssertionError: imo-bench-algebra-036
E           assert 'Functional Equation' in {'Algebra', 'Combinatorics', 'Geometry', 'Number theory'}
3 failed

QA answer-key check (every shipped README sample carries a real answer key):

FAIL (0.0s): imo-bench-algebra-036: gold is an answer, not a category label
math_with_autograder: 0/1 passed

After the fix

This branch, same commands:

{'problem_id': 'imo-bench-algebra-036', 'expected_answer': '$Y(x)=A+\\frac{B}{x}-x$', 'category': 'Algebra', 'subcategory': 'Functional Equation', 'source': 'Iran 2002'}
'frac{1}{a}\\right)+a Y(a)+\\frac{b}{a}\n\\]\n'

tests/unit_tests/test_imo_answerbench_prepare.py ...    3 passed

prepare() against the real pinned CSV: 400 rows, 100 per category, and only imo-bench-algebra-036 differs from main's output.

Live run on one H100 (Qwen3-8B policy on vLLM, openai/gpt-oss-20b judge on NIM, README's gym eval run --agent math_with_autograder_simple_agent --input .../example.jsonl --num-repeats 1). The algebra-036 prompt no longer contains the answer. The model's own answer (Y(x) = -x + c) went to the judge against the real gold, and the judge rejected it for missing solutions. Verifier controls on the same row:

shipped/1/reference  \boxed{$Y(x)=A+\frac{B}{x}-x$}  -> reward 1.0 (math-verify, no judge call)
shipped/1/wrong      \boxed{-987654321}               -> reward 0.0 (one judge call)

Before the fix the reference control could only box Algebra, so a correct answer could never score.

Validation

• QA testcase math_with_autograder (internal suite, one H100): main 0/2, where the answer-key case fails on this row and the live case fails on a stale QA-side agent_ref expectation (fixed in the suite, not in Gym); this branch 2/2 with the suite's matching update
• tests/unit_tests/test_imo_answerbench_prepare.py: 3 failed before, 3 passed after
• tests/unit_tests/test_benchmarks.py, test_task_data.py, test_train_data_utils.py: 520 passed; test_every_repo_benchmark_appears_in_listing fails identically on main on the test machine (DNS: apex_shortlist/hle opencode/pi configs cannot resolve), unrelated
• gym env test +entrypoint=resources_servers/math_with_autograder +should_validate_data=true: 25 passed, example data validated
• pre-commit run --all-files: passed, no changes
• Rollout evidence: the live run above (5 rows, Qwen3-8B + gpt-oss-20b judge): rewards [0, 0, 1, 0, 1]; algebra-036 graded against the real gold

Compatibility

Benchmark results change for one of 400 IMO AnswerBench problems: algebra-036 is now gradable (it could only score 0 before) and no longer shows the answer in the prompt. Gym and NeMo Skills now differ on that row until Skills gets the same repair or superhuman#13 is fixed. A future pin to a CSV with any other misaligned row now fails prepare loudly instead of writing it.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[qiaochuz-nv]

/ok to test f909cdd

[cmunley1]

lgtm, @gwarmstrong can you take a look?

[gwarmstrong]

lgtm as well, thanks for the fix

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39457996667