Repository navigation
fix(benchmarks): read Yes/No verdicts as whole words in pope, hallusion_bench, drivel_binary - #1816
Merged
Merged
Conversation
…on_bench, drivel_binary These three adapters scored a sample with `reference in filtered_prediction.strip().upper()`, the same substring bug fixed for HaluEval in #1800. For a "NO" label, any word containing the letters "no" ("know", "not", "now", "no explanation", ...) credited a "Yes" or verdict-less reply as correct, inflating accuracy and, via the acc-derived prediction, precision/recall/ f1/yes_ratio. hallusion_bench feeds acc straight into aAcc/fAcc/qAcc, so its primary metric was inflated too. - Promote extract_verdict() to a shared leaf util evalscope/utils/yes_no.py and reuse it in halu_eval (behaviour unchanged) and the three adapters: credit a sample only when the reply carries exactly one whole-word verdict equal to the reference. - Bump evaluation_version for the scoring-semantics change: pope v1.0->v1.1, drivel_binary v1.0->v1.1, hallusion_bench v1.1->v1.2. halu_eval stays v1.1 (pure refactor). - Add tests/utils/test_yes_no.py and tests/benchmark/test_yes_no_verdict_scoring.py.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
POPE and Drivelology still fabricate predicted classes for ambiguous responses during aggregation, corrupting diagnostic metrics.
2 open findings
What changed in this PR
Centralizes strict whole-word Yes/No verdict extraction and applies it across affected benchmark adapters.
Changes:
- Adds a shared
extract_verdict()utility. - Updates scoring semantics and evaluation versions for three benchmarks.
- Adds utility and cross-adapter regression tests.
| File | Description |
|---|---|
evalscope/utils/yes_no.py |
Adds shared verdict extraction. |
evalscope/benchmarks/pope/pope_adapter.py |
Uses strict verdict scoring and bumps version. |
evalscope/benchmarks/hallusion_bench/hallusion_bench_adapter.py |
Uses strict verdict scoring and bumps version. |
evalscope/benchmarks/drivelology/drivelology_binary_adapter.py |
Uses strict verdict scoring and bumps version. |
evalscope/benchmarks/halu_eval/halu_eval_adapter.py |
Reuses the shared utility. |
tests/utils/test_yes_no.py |
Tests verdict extraction behavior. |
tests/benchmark/test_yes_no_verdict_scoring.py |
Tests per-sample adapter scoring. |
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
4 of 9 tasks
Yunnglin
pushed a commit
that referenced
this pull request
Oct 22, 2026
…#1912) aggregate_scores rebuilt each prediction from acc: a reply with no verdict (or both) on a "No" item became a "Yes", so one "Yes" and one "I do not know." reported yes_ratio 1.0 and precision 0.5. #1816 fixed the same pattern in pope and drivel_binary by reading the verdict with extract_verdict; do the same here and bump evaluation_version to v1.2. Accuracy is unchanged. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 22, 2026
Yunnglin
pushed a commit
that referenced
this pull request
Oct 23, 2026
match_score credited a reply when the gold label appeared anywhere in it as a word, so "YES or NO: MAYBE" scored 1 against every label, and aggregate_scores read the predicted class with substring checks, so "I cannot tell." became a NO prediction and lowered NO precision. Read the verdict the way evalscope.utils.yes_no does for Yes/No benchmarks (#1800, #1816): exactly one label, written YES or Yes, or a bare lowercase label. Bump evaluation_version to v1.2. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Yunnglin
pushed a commit
that referenced
this pull request
Oct 26, 2026
… and in the report (#1934) match_score credited a reply when the uppercase label was a substring of the answer line, while aggregate_scores, which produces the reported accuracy, required the line to equal YES or NO after upper-casing. The two disagreed on ordinary replies: 'ANSWER: No' scored 0 per sample but counted as correct in the report, and 'ANSWER: NO.' or 'ANSWER: **YES**.' scored 1 per sample but were wrong in the report, so four correct replies reported accuracy 0.5. Read the verdict with extract_verdict in both places, as #1816 did for pope, hallusion_bench and drivel_binary, and bump evaluation_version to v1.1. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Follow-up to #1800.
pope,hallusion_benchanddrivel_binaryscored each sample with the same substring check #1800 removed from HaluEval:For a
NOlabel, any reply whose uppercased text contains the letters "no" (know,not,now,nonsense,no explanation, ...) is credited as correct even when the model actually said "Yes" or gave no verdict:Yes, I know it.I do not know.Yes, no explanation needed.popeanddrivel_binaryderive the predicted class fromaccinaggregate_scores, soprecision/recall/f1_score/yes_ratiowere affected too.hallusion_benchfeedsaccstraight intoaAcc/fAcc/qAcc(a group counts only when all its samples do), so an inflatedaccalso inflated its primary metric.How
extract_verdict()from fix(halueval): read Yes/No verdicts as whole words, not substrings #1800 into a shared leaf utilevalscope/utils/yes_no.py, reused byhalu_eval(behaviour unchanged) and the three adapters: credit a sample only when the reply carries exactly one whole-word verdict (Yes/No/YES/NO) equal to the reference; both verdicts or neither score 0.evalscope/utils/already hosts cross-benchmark answer matching (multi_choices.py), and the helper is purere, solint-importsneeds no new exception.evaluation_versionfor the scoring-semantics change (AGENTS.md):popev1.0 -> v1.1,drivel_binaryv1.0 -> v1.1,hallusion_benchv1.1 -> v1.2;halu_evalstays v1.1 (pure refactor). NoBenchmarkMeta.descriptionchange, so no_meta/docs regeneration.Tests
tests/utils/test_yes_no.py: unit tests for the sharedextract_verdict.tests/benchmark/test_yes_no_verdict_scoring.py: cross-adapter regressions through the realcalculate_metricspath.AI assistance: drafted with an AI coding assistant and verified locally with the tests above.