Repository navigation
fix: enforce strict judge output parsing - #1588
Merged
Merged
Conversation
… 0.0 sentinel
Builds on the strict parsing in this branch with three corrections found while
reviewing it against the upstream reference implementations.
AIME: strict whole-response equality also rejects formats the judge prompt itself
demonstrates. Four of its eight few-shot examples show "No\n(these are actually
equal, ...)", so a trailing parenthetical is taught behaviour, not malformed
output; punctuation ("Yes.") and markdown ("**Yes**") are likewise common. Require
the verdict to occupy a line of its own instead, which still rejects the reported
red cases ("Yes, the answer is incorrect.") and additionally flags a response
carrying both a bare Yes and a bare No line. Extracted as a staticmethod so the
parse is unit-testable without stubbing the adapter.
AIR-Bench, parsing: anchoring re.fullmatch to the whole response rejects
"8 9\n<explanation>", which is the most common shape a GPT-4-class judge produces
for this prompt, plus comma-separated and leading-blank-line variants. Anchor to
the first non-empty line instead. The official cal_score.py is equally strict, but
it post-processes recorded GPT-4 output offline whereas evalscope parses whatever
judge the user configures.
AIR-Bench, failure handling: reporting judge_score=0.0 puts an out-of-scale value
(the rating scale starts at 1) into the default mean aggregation, so a parse
failure reads as a confident low rating. Omitting the metric keys drops the sample
from both numerator and denominator, matching the official script's fail_num
semantics, and AggScore.num already exposes the surviving count. This matters more
after the parser was tightened, because parse failures stopped being rare.
Scope is limited to the two adapters named in #1578; the partial-pass semantics of
do_swap are left unchanged.
Refs #1578
Contributor
There was a problem hiding this comment.
Pull request overview
Tightens LLM-judge output parsing for the AIME and AIR-Bench Chat benchmark adapters so only responses matching the requested strict formats can influence reported metrics, addressing the parsing vulnerabilities described in #1578.
Changes:
- Enforce line-anchored parsing for AIME judge verdicts (“Yes”/“No”) and mark malformed judge outputs via
parse_failed. - Enforce first-non-empty-line strict two-score parsing for AIR-Bench Chat and omit metrics entirely when no valid judge pass can be parsed (with
parse_failedmetadata for diagnostics). - Add regression tests covering strict parsing behavior and parse-failure metadata.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| tests/benchmark/test_judge_output_parsing.py | Adds regression tests for strict judge-output parsing and parse-failure handling for AIME and AIR-Bench Chat. |
| evalscope/benchmarks/air_bench/air_bench_chat_adapter.py | Anchors judge score extraction to the first non-empty line and improves parse-failure tracking/metric omission behavior. |
| evalscope/benchmarks/aime/aime_adapter.py | Replaces substring-based verdict detection with strict per-line verdict parsing and parse_failed signaling. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+185
to
187
| judge_verdicts = self._parse_judge_verdicts(judge_response) | ||
| is_correct = judge_verdicts == {'yes'} | ||
| score.value = { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
I require exact judge-output formats for AIME and AIR-Bench Chat before their results affect reported metrics.
Why
Free-text substring and positional-number parsing let verbose or quoted judge content change benchmark scores.
Verification
python -m pytest tests/benchmark/test_judge_output_parsing.py -qmake lintFixes #1578