HLE Benchmark

benchmarks/hle/README.md

@@ -0,0 +1,61 @@
+# HLE Benchmark
+
+Benchmark wrapper for [Humanity's Last Exam](https://huggingface.co/datasets/cais/hle), a
+2158-question (text-only subset) exam covering graduate-level STEM and humanities knowledge.
+
+- **Tasks**: 2158 text-only questions (image questions filtered at prepare time)
+- **Reward**: binary; LLM judge checks whether the model's response matches the ground-truth answer
+- **Metrics**: `pass@1/judge_accuracy` — fraction of questions judged correct
+
+The judge uses the official HLE evaluation prompt adapted from
+[`centerforaisafety/hle`](https://github.com/centerforaisafety/hle), which extracts the model's
+final answer and checks it against the expected answer with a yes/no verdict. The policy model
+serves as the judge — no separate judge server is needed.
+
+## Dataset access
+
+`cais/hle` is a gated HuggingFace dataset. Request access at
+[https://huggingface.co/datasets/cais/hle](https://huggingface.co/datasets/cais/hle), then
+authenticate:
+
+```bash
+huggingface-cli login
+```
+
+## Prepare benchmark data
+
+```bash
+ng_prepare_benchmark "+config_paths=[benchmarks/hle/config.yaml]"
+```
+
+Downloads `cais/hle`, filters to text-only questions, and writes
+`benchmarks/hle/data/hle_benchmark.jsonl`.
+
+## Running servers
+
+```bash
+config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
+benchmarks/hle/config.yaml"
+ng_run "+config_paths=[$config_paths]"
+```
+
+Requires `policy_base_url` / `policy_api_key` / `policy_model_name` in
+`env.yaml` (or passed as CLI overrides).
+
+## Collect rollouts
+
+```bash
+ng_collect_rollouts \
+    +agent_name=hle_equivalence_llm_judge_simple_agent \
+    +input_jsonl_fpath=benchmarks/hle/data/hle_benchmark.jsonl \
+    +output_jsonl_fpath=results/hle_rollouts.jsonl \
+    +prompt_config=benchmarks/hle/prompts/default.yaml \
+    +num_repeats=1 \
+    "++responses_create_params={temperature: 0.0}"
+```
+
+Use `temperature: 0.0` to match the nemo-skills evaluation setup and ensure reproducible scores.
+
+## Metrics
+
+`pass@1/judge_accuracy` is the headline metric.

benchmarks/hle/config.yaml

@@ -0,0 +1,37 @@
+# Chain to the equivalence_llm_judge resources server config
+config_paths:
+  - resources_servers/equivalence_llm_judge/configs/equivalence_llm_judge.yaml
+
+# HLE resources server: pure LLM judge, no symbolic verification step.
+# Uses the official HLE judge prompt from centerforaisafety/hle.
+hle_equivalence_llm_judge_resources_server:
+  _inherit_from: equivalence_llm_judge
+  resources_servers:
+    equivalence_llm_judge:
+      judge_model_server:
+        type: responses_api_models
+        name: policy_model
+      judge_responses_create_params:
+        input: []
+      judge_system_message: ""
+      judge_equal_label: "Judgement: yes"
+      judge_not_equal_label: "Judgement: no"
+      judge_prompt_template_fpath: ../../benchmarks/hle/prompts/judge.txt
+      response_extract_regex: null
+      question_extract_regex: null
+      check_twice_swap: false
+      use_per_record_regex: false
+
+hle_equivalence_llm_judge_simple_agent:
+  _inherit_from: equivalence_llm_judge_simple_agent
+  responses_api_agents:
+    simple_agent:
+      resources_server:
+        name: hle_equivalence_llm_judge_resources_server
+      datasets:
+      - name: hle
+        type: benchmark
+        jsonl_fpath: benchmarks/hle/data/hle_benchmark.jsonl
+        prompt_config: benchmarks/hle/prompts/default.yaml
+        prepare_script: benchmarks/hle/prepare.py
+        num_repeats: 1

benchmarks/hle/data/.gitignore

@@ -0,0 +1 @@
+*.jsonl

benchmarks/hle/prepare.py

@@ -0,0 +1,73 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Prepare HLE evaluation data for NeMo Gym.
+
+Downloads Humanity's Last Exam (HLE) from HuggingFace and converts to Gym JSONL
+format compatible with the math_with_judge resource server.
+
+Only text-only questions are included (image questions are filtered out).
+"""
+
+from pathlib import Path
+
+from nemo_gym.global_config import HF_TOKEN_KEY_NAME, get_global_config_dict
+
+
+BENCHMARK_DIR = Path(__file__).parent
+DATA_DIR = BENCHMARK_DIR / "data"
+OUTPUT_FPATH = DATA_DIR / "hle_benchmark.jsonl"
+
+
+def prepare() -> Path:
+    """Download HLE test data and convert to Gym JSONL format."""
+    import json
+
+    from datasets import load_dataset
+
+    print("Downloading HLE from HuggingFace...")
+    hf_token = get_global_config_dict().get(HF_TOKEN_KEY_NAME)
+    ds = load_dataset("cais/hle", split="test", token=hf_token)
+
+    DATA_DIR.mkdir(parents=True, exist_ok=True)
+
+    # Use arrow format to avoid decoding the Image column, which requires Pillow.
+    # In arrow format the image column is a raw string: empty for text-only questions,
+    # base64 data for image questions.
+    rows = []
+    skipped_image = 0
+    for batch in ds.with_format("arrow").iter(batch_size=500):
+        for i in range(batch.num_rows):
+            if batch.column("image")[i].as_py():
+                skipped_image += 1
+                continue
+
+            row = {
+                "question": batch.column("question")[i].as_py(),
+                "expected_answer": batch.column("answer")[i].as_py(),
+                "answer_type": batch.column("answer_type")[i].as_py(),  # not used for grading; useful for analysis
+                "uuid": batch.column("id")[i].as_py(),
+            }
+            rows.append(json.dumps(row) + "\n")
+
+    with open(OUTPUT_FPATH, "w") as f:
+        f.writelines(rows)
+
+    print(f"Wrote {len(rows)} problems to {OUTPUT_FPATH} (skipped {skipped_image} image questions)")
+    return OUTPUT_FPATH
+
+
+if __name__ == "__main__":
+    prepare()

benchmarks/hle/prompts/default.yaml

@@ -0,0 +1,7 @@
+system: |-
+  Your response should be in the following format:
+  Explanation: {{your explanation for your final answer}}
+  Answer: {{your succinct, final answer}}
+  Confidence: {{your confidence score between 0% and 100% for your answer}}
+
+user: "{question}"

benchmarks/hle/prompts/judge.txt

@@ -0,0 +1,17 @@
+Judge whether the following [response] to [question] is correct or not based on the precise and unambiguous [correct_answer] below.
+
+[question]: {question}
+
+[response]: {generated_answer}
+
+Your judgement must be in the format and criteria specified below:
+
+extracted_final_answer: The final exact answer extracted from the [response]. Put the extracted answer as 'None' if there is no exact, final answer to extract from the response.
+
+[correct_answer]: {expected_answer}
+
+Reasoning: Explain why the extracted_final_answer is correct or incorrect based on [correct_answer], focusing only on if there are meaningful differences between [correct_answer] and the extracted_final_answer. Do not comment on any background to the problem, do not attempt to solve the problem, do not argue for any answer different than [correct_answer], focus only on whether the answers match.
+
+Judgement: Answer 'yes' if extracted_final_answer matches the [correct_answer] given above, or is within a small margin of error for numerical problems. Answer 'no' otherwise, i.e. if there if there is any inconsistency, ambiguity, non-equivalency, or if the extracted answer is incorrect.
+
+Confidence: The extracted confidence score between 0% and 100% from [response]. Put 100 if there is no confidence score available.