HLE Benchmark by fsiino-nvidia · Pull Request #1028 · NVIDIA-NeMo/Gym

fsiino-nvidia · 2026-04-08T18:16:51Z

wandb: https://wandb.ai/nvidia/fsiino-gym-dev/runs/nj1codxo

Key metrics for hle_math_with_judge_simple_agent:
{
    "mean/reward": 0.2015755329008341,
    "mean/input_tokens": 335.064874884152,
    "mean/output_tokens": 31830.81278962002,
    "mean/total_tokens": 32165.87766450417
}

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

copy-pr-bot · 2026-04-08T18:16:54Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

…swer Signed-off-by: Frankie Siino <fsiino@nvidia.com>

copy-pr-bot · 2026-04-21T00:06:41Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

gwarmstrong

Thanks Frankie! The new flags on math_with_judge are a clean extension, defaults preserve prior behavior, and the metrics in the PR body look reasonable. A few asks before merge, most are inline, a few general ones here:

Blockers

Please add benchmarks/hle/README.md with the ng_prepare_benchmark + ng_run + ng_collect_rollouts quickstart. Peer benchmarks (omniscience, birdbench, mrcr) all ship one. While you're there, add an empty __init__.py to match.
Please add tests for the 7 new math_with_judge flags — tests/test_app.py isn't modified. (Detail in the inline comment on app.py.)
See inline: drop the system: block regression in prompts/default.yaml, and move the inlined judge prompt out of config.yaml to a template file (pattern from equivalence_llm_judge).

Non-blocking

With skip_symbolic + judge_use_full_response + judge_check_position_bias=false, HLE is effectively LLM-judge-only — same contract as equivalence_llm_judge. Worth considering whether HLE belongs there, or whether the 7 new fields could collapse (e.g. JudgePromptSpec + judge_mode enum). Happy to defer.

gwarmstrong · 2026-04-22T22:29:26Z

@@ -0,0 +1 @@
+user: "{question}"


Skills' nemo_skills/prompt/config/generic/hle.yaml has a system: block that forces the Explanation: ... / Answer: ... / Confidence: ... format. This file drops the system message entirely. That's a deviation from both the Skills baseline and the official HLE eval — and the judge prompt downstream expects extracted_final_answer and Confidence: fields, so I'd expect it to move the numbers.

Commit message says "bare question format" — if this is intentional, can you add a one-line note to the PR body explaining why?

gwarmstrong · 2026-04-22T22:29:34Z

+      judge_not_equal_label: "Judgement: no"
+      judge_prompt_template: |-
+        Judge whether the following [response] to [question] is correct or not based on the precise and unambiguous [correct_answer] below.
+
+        [question]: {question}
+
+        [response]: {first_answer}
+
+        Your judgement must be in the format and criteria specified below:
+
+        extracted_final_answer: The final exact answer extracted from the [response]. Put the extracted answer as 'None' if there is no exact, final answer to extract from the response.
+
+        [correct_answer]: {second_answer}
+
+        Reasoning: Explain why the extracted_final_answer is correct or incorrect based on [correct_answer], focusing only on if there are meaningful differences between [correct_answer] and the extracted_final_answer. Do not comment on any background to the problem, do not attempt to solve the problem, do not argue for any answer different than [correct_answer], focus only on whether the answers match.
+
+        Judgement: Answer 'yes' if extracted_final_answer matches the [correct_answer] given above, or is within a small margin of error for numerical problems. Answer 'no' otherwise, i.e. if there if there is any inconsistency, ambiguity, non-equivalency, or if the extracted answer is incorrect.


The judge prompt is a ~17-line |- literal inside YAML. resources_servers/equivalence_llm_judge uses the cleaner pattern:

# configs/equivalence_llm_judge.yaml:10 judge_prompt_template_fpath: prompt_templates/equivalence_llm_judge.txt

Loaded in model_post_init (see equivalence_llm_judge/app.py:266-267). Benefits: diffable as prose, no YAML-escape gymnastics, shareable across benchmarks.

Suggested change:

Add judge_prompt_template_fpath: Optional[str] = None to LibraryJudgeMathResourcesServerConfig.

Wire it into _generate_judge_evaluation the same way equivalence_llm_judge does.

Move the HLE rubric to benchmarks/hle/prompts/judge.txt and reference it here.

The existing judge_prompt_template: str can stay as a fallback — this is strictly additive.

gwarmstrong · 2026-04-22T22:29:42Z

+    # base64 data for image questions.
+    rows = []
+    skipped_image = 0
+    for batch in ds.with_format("arrow").iter(batch_size=500):
+        for i in range(batch.num_rows):
+            if batch.column("image")[i].as_py():


Non-blocking: Skills' prepare preserves subset_for_metrics, category, raw_subject, reference_solution, author_name, canary. This version keeps only question, expected_answer, answer_type, uuid.

category / subset_for_metrics in particular enable per-category accuracy (a Skills convention). If per-category breakdown isn't planned, fine to leave; otherwise worth preserving now since it's ~4 lines.

gwarmstrong · 2026-04-22T22:29:52Z

+    skip_symbolic: bool = False  # when True, bypass symbolic comparison and always use the judge
+    judge_system_message: Optional[str] = None  # overrides JUDGE_SYSTEM_MESSAGE; empty string = no system message
+    judge_prompt_template: Optional[str] = (
+        None  # overrides JUDGE_PROMPT_TEMPLATE; uses {question}, {first_answer}, {second_answer}
+    )
+    judge_equal_label: Optional[str] = None  # overrides JUDGE_EQUAL_LABEL
+    judge_not_equal_label: Optional[str] = None  # overrides JUDGE_NOT_EQUAL_LABEL
+    judge_check_position_bias: bool = True  # when False, makes a single judge call instead of two
+    judge_use_full_response: bool = False  # when True, always pass the full model response to the judge instead of the math_verify extracted answer


Non-blocking — 7 new fields is a lot. Specifically, skip_symbolic=true + judge_use_full_response=true + judge_check_position_bias=false together reproduce what resources_servers/equivalence_llm_judge already does.

Two questions:

Was routing HLE through equivalence_llm_judge considered? If symbolic_accuracy reporting alongside judge_accuracy is the reason to go through math_with_judge, please call it out in the PR body.

If the 7 fields stay, a follow-up could collapse them: the four judge-text fields (judge_prompt_template, judge_system_message, judge_equal_label, judge_not_equal_label) co-vary — one JudgePromptSpec loaded from a file covers them all. The two symbolic booleans (should_use_judge, skip_symbolic) could collapse into judge_mode: symbolic_first | judge_only | symbolic_only.

Happy to defer. Also — please add test coverage for the new flags (tests/test_app.py isn't touched): at minimum skip_symbolic=True, judge_check_position_bias=False, judge_use_full_response=True, and one parametrized override test for the four judge-text fields. Coverage bar is 95% per Gym CLAUDE.md.

gwarmstrong · 2026-04-22T22:29:57Z

    judge_responses_create_params: NeMoGymResponseCreateParamsNonStreaming
    should_use_judge: bool = True
+    skip_symbolic: bool = False  # when True, bypass symbolic comparison and always use the judge
+    judge_system_message: Optional[str] = None  # overrides JUDGE_SYSTEM_MESSAGE; empty string = no system message


Nit: the code at lines ~271-275 treats None → class default, "" → no system role (via the truthy check downstream). The HLE config relies on "" meaning "skip system". It works, but None vs "" semantics aren't obvious — the inline comment here says "empty string = no system message" which is good; consider adding the same note to the field's defaults in configs/math_with_judge.yaml so users configuring it from YAML don't have to read the source.

fsiino-nvidia · 2026-04-23T21:15:41Z

Thank you for the thorough review here @gwarmstrong !

I agree that equivalence_llm_judge is very likely the better fit. I had gone with math_with_judge largely because HLE has math questions, and the symbolic verification stage seemed useful. It became clear after running the benchmark that we needed to bypass symbolics entirely and use judge_correct as the key metric. At that point it had to be configured to behave like a general-purpose judge.

I'm happy to switch to equivalence_llm_judge though. I'll take a look at what that migration looks like and update the PR.

- use equivalence_llm_judge for HLE - revert math_with_judge - add readme and empty __init__ - restore system: block in prompt to match skills Signed-off-by: Frankie Siino <fsiino@nvidia.com>

…ing parser Drops the opt-in skills_parity_mode pipeline (_strip_think_tags, _verify_answer_skills_parity, prefill shortcuts, always-judge mode) that was layered on top of the standard symbolic-first cascade to mirror Skills' per-rollout judge behavior. It worked — parity@1 tightened to −0.75 pp against Skills' gemini-2.5-pro baseline — but it also: * hand-rolled reasoning-trace stripping that the vLLM server can do cleaner at the model edge via `--reasoning-parser <name>`, * overloaded the math_with_judge cascade with four new knobs when the same file is already being extended by other migrations (e.g. HLE PR NVIDIA-NeMo#1028 adds skip_symbolic / judge_use_full_response / etc.). Simpler path: rely on the vLLM reasoning parser to strip <think>...</think> at the model edge, and keep the symbolic-first cascade + raw-\boxed-to- judge fix (commit prior to this one) as the judge-input hardening. The residual parity bias (~+1.7 pp, Gym slightly higher than Skills) comes from Gym never having the chance to demote a symbolically-correct answer that Skills' always-judge pipeline would flag as wrong. Inside the 5 pp LLM-judge threshold; acceptable. Also adds a one-line reasoning-parser reminder to the benchmark README. Reverts: a421c5f (upstream SHA prior to rebase; current commit 2e4157a on branch). Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

…/bidirectional knobs Ports nemo_skills.evaluation.metrics.utils.is_correct_judgement to math_with_judge so its verdict parser natively recognises the three common judge output formats without per-benchmark overrides: * Arena-Hard "[[A=B]]" / "[[A!=B]]" (ClassVar default pair) * Math-judge "Judgement: Yes" / "Judgement: No" (fallback; case-insensitive; tolerates **bold** around the label or the verdict — covers the gpt-4o rendering) * Autograder "\\boxed{Correct}" / "\\boxed{Incorrect}" (fallback; case-insensitive) The configured judge_equal_label / judge_not_equal_label still take precedence when they appear in the output (existing HLE-style callers keep working). Only when neither configured label is present in the judge's response does the parser consult the fallback formats. Drops three now-redundant config fields from benchmarks/imo_answerbench/config.yaml: * judge_equal_label: "\\boxed{Correct}" -> parser fallback handles it * judge_not_equal_label: "\\boxed{Incorrect}" -> parser fallback handles it * judge_bidirectional: false -> accept the extra judge call per rollout; the IMO prompt is symmetric enough that "is gold equivalent to model?" and "is model equivalent to gold?" give the same verdict, and keeping bidirectional reuses the existing shared code path. Motivation: IMO AnswerBench and HLE (PR NVIDIA-NeMo#1028) were both independently adding overlapping knobs to math_with_judge to teach it new judge output formats (one call for IMO's \boxed, one for HLE's lowercase Judgement: yes). Centralising that in the parser keeps math_with_judge lean — benchmark authors provide a prompt via `judge_prompt_path` and the server does the rest. After this commit benchmarks/imo_answerbench/config.yaml has exactly ONE math_with_judge extension beyond the base: `judge_prompt_path: benchmarks/imo_answerbench/prompts/judge.yaml`. Every other pipeline difference (reasoning-trace stripping, symbolic-first cascade, bidirectional judging) is now shared with aime24 / aime25 / dapo17k. Tests: 10 new in TestMultiFormatVerdictParser + existing 30 still pass. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

…ing parser Drops the opt-in skills_parity_mode pipeline (_strip_think_tags, _verify_answer_skills_parity, prefill shortcuts, always-judge mode) that was layered on top of the standard symbolic-first cascade to mirror Skills' per-rollout judge behavior. It worked — parity@1 tightened to −0.75 pp against Skills' gemini-2.5-pro baseline — but it also: * hand-rolled reasoning-trace stripping that the vLLM server can do cleaner at the model edge via `--reasoning-parser <name>`, * overloaded the math_with_judge cascade with four new knobs when the same file is already being extended by other migrations (e.g. HLE PR NVIDIA-NeMo#1028 adds skip_symbolic / judge_use_full_response / etc.). Simpler path: rely on the vLLM reasoning parser to strip <think>...</think> at the model edge, and keep the symbolic-first cascade + raw-\boxed-to- judge fix (commit prior to this one) as the judge-input hardening. The residual parity bias (~+1.7 pp, Gym slightly higher than Skills) comes from Gym never having the chance to demote a symbolically-correct answer that Skills' always-judge pipeline would flag as wrong. Inside the 5 pp LLM-judge threshold; acceptable. Also adds a one-line reasoning-parser reminder to the benchmark README. Reverts: a421c5f (upstream SHA prior to rebase; current commit 2e4157a on branch). Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

…/bidirectional knobs Ports nemo_skills.evaluation.metrics.utils.is_correct_judgement to math_with_judge so its verdict parser natively recognises the three common judge output formats without per-benchmark overrides: * Arena-Hard "[[A=B]]" / "[[A!=B]]" (ClassVar default pair) * Math-judge "Judgement: Yes" / "Judgement: No" (fallback; case-insensitive; tolerates **bold** around the label or the verdict — covers the gpt-4o rendering) * Autograder "\\boxed{Correct}" / "\\boxed{Incorrect}" (fallback; case-insensitive) The configured judge_equal_label / judge_not_equal_label still take precedence when they appear in the output (existing HLE-style callers keep working). Only when neither configured label is present in the judge's response does the parser consult the fallback formats. Drops three now-redundant config fields from benchmarks/imo_answerbench/config.yaml: * judge_equal_label: "\\boxed{Correct}" -> parser fallback handles it * judge_not_equal_label: "\\boxed{Incorrect}" -> parser fallback handles it * judge_bidirectional: false -> accept the extra judge call per rollout; the IMO prompt is symmetric enough that "is gold equivalent to model?" and "is model equivalent to gold?" give the same verdict, and keeping bidirectional reuses the existing shared code path. Motivation: IMO AnswerBench and HLE (PR NVIDIA-NeMo#1028) were both independently adding overlapping knobs to math_with_judge to teach it new judge output formats (one call for IMO's \boxed, one for HLE's lowercase Judgement: yes). Centralising that in the parser keeps math_with_judge lean — benchmark authors provide a prompt via `judge_prompt_path` and the server does the rest. After this commit benchmarks/imo_answerbench/config.yaml has exactly ONE math_with_judge extension beyond the base: `judge_prompt_path: benchmarks/imo_answerbench/prompts/judge.yaml`. Every other pipeline difference (reasoning-trace stripping, symbolic-first cascade, bidirectional judging) is now shared with aime24 / aime25 / dapo17k. Tests: 10 new in TestMultiFormatVerdictParser + existing 30 still pass. Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

gwarmstrong · 2026-04-28T19:03:05Z

@fsiino-nvidia fwiw the best approach is probalby to make a new subclass of whichever resources_server works best for you and add the specific judging logic you need there. This is what I am going to do for IMO-answerbench, PR here #1155 (though still working on the subclass port). This will help make sure we minimize disruptions to existing workflows, especially ones that are used heavily.

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia · 2026-05-05T00:28:20Z

@gwarmstrong sorry for the delay here. Was caught in some other threads for a bit. This now uses equivalence_llm_judge. Also noted on the subclass moving forward. In this case there was no custom logic needed for the resource server.

Updated score is in the edited description above. There was a ~2% drop, but still well within the expected number with this checkpoint (3/5 step 1900).

gwarmstrong

@fsiino-nvidia LGTM as long as you verify the numbers match a reference implementation

gwarmstrong

actually, please make sure the README.md examples are runnable by copy paste. I believe you need to add +prompt_config=benchmarks/hle/prompts/default.yaml to ng_collect_rollouts for a proper run.

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia · 2026-05-05T21:13:35Z

.. LGTM as long as you verify the numbers match a reference implementation

Confirmed. Expected mean/reward is 19.97.

actually, please make sure the README.md examples are runnable by copy paste. I believe you need to add +prompt_config=benchmarks/hle/prompts/default.yaml to ng_collect_rollouts for a proper run.

Added in 0b159dd

Init hle - data preparation + prompt

9e6b579

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia added 5 commits April 8, 2026 14:38

Handle image casting

94be480

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Use with_format instead of cast_column

592217b

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Toggle judge fallback in math_with_judge resource server

e5aedbb

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

use official hle judge prompt and bare question format

e74768e

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Pass full model response to judge instead of math_verify extracted an…

0525c81

…swer Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia marked this pull request as ready for review April 21, 2026 00:12

Merge remote-tracking branch 'github/main' into fsiino/hle-benchmark

e18652a

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia requested review from bxyu-nvidia and gwarmstrong April 21, 2026 00:59

gwarmstrong reviewed Apr 22, 2026

View reviewed changes

Migrate HLE benchmark to equivalence_llm_judge

105c510

- use equivalence_llm_judge for HLE - revert math_with_judge - add readme and empty __init__ - restore system: block in prompt to match skills Signed-off-by: Frankie Siino <fsiino@nvidia.com>

fsiino-nvidia added 3 commits May 4, 2026 14:30

Fix prompt template curly brace escaping

3cd34b5

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Use working directory for judge template

478f9f4

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Merge remote-tracking branch 'github/main' into fsiino/hle-benchmark

51a4e84

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

Merge branch 'main' into fsiino/hle-benchmark

08de775

gwarmstrong approved these changes May 5, 2026

View reviewed changes

gwarmstrong requested changes May 5, 2026

View reviewed changes

Add prompt_config arg to readme

0b159dd

Signed-off-by: Frankie Siino <fsiino@nvidia.com>

gwarmstrong approved these changes May 5, 2026

View reviewed changes

gwarmstrong merged commit 70588e7 into main May 5, 2026
5 checks passed

gwarmstrong deleted the fsiino/hle-benchmark branch May 5, 2026 21:23

qiaochuz-nv mentioned this pull request Oct 12, 2026

[NVBug 6900619] Fix column-shifted imo-bench-algebra-036 row in math_with_autograder data #4506

Closed

Conversation

fsiino-nvidia commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot Bot commented Apr 8, 2026

Uh oh!

copy-pr-bot Bot commented Apr 21, 2026

Uh oh!

gwarmstrong left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gwarmstrong Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

gwarmstrong Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

gwarmstrong Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

gwarmstrong Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

gwarmstrong Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

fsiino-nvidia commented Apr 23, 2026

Uh oh!

gwarmstrong commented Apr 28, 2026

Uh oh!

fsiino-nvidia commented May 5, 2026

Uh oh!

gwarmstrong left a comment

Choose a reason for hiding this comment

Uh oh!

gwarmstrong left a comment

Choose a reason for hiding this comment

Uh oh!

fsiino-nvidia commented May 5, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fsiino-nvidia commented Apr 8, 2026 •

edited

Loading

gwarmstrong left a comment •

edited

Loading