PubMedQA: unrecognized answers inflate recall and macro F1, reversing model rankings #1827

Self-Check List

• I have read the relevant native-evaluation, dataset and scoring parameters documentation.
• I have reviewed the FAQ, including its recommendation to check the latest main branch.
• I have searched open/closed issues and PRs for PubMedQA, F1, recall, abstention and unparseable responses; no matching report found. I refreshed the PubMedQA search before filing.

Problem Description

PubMedQA's default rule-based scorer counts an unrecognized answer as wrong for accuracy, but silently drops it from the denominator used for recall and macro F1. A model that leaves questions unanswered can therefore receive perfect recall/F1, and the reported F1 ranking can reverse.

Here is an offline replay on the same 30 synthetic cases (10 YES, 10 NO, 10 MAYBE), using canned completions through the actual record conversion, calculate_metrics and aggregate_scores methods:

For each of A's three classes, TP=8, FP=0 and FN=2. Under the adapter's existing three-class macro convention, recall is 8/10 and F1 is 16/18. Instead, the implementation computes recall as 8/8. The apparent F1 lead of 10 percentage points becomes a deficit of 1.1111 points when corrected.

These are controlled synthetic responses, not measurements of commercial models. There are no API failures or judge failures: every sample has a successful rule-scoring status. Empty responses reproduce the same denominator problem. The demonstrated scope is PubMedQA recall/F1; the primary accuracy metric remains correct.

EvalScope Version (Required)

• Latest published PyPI package 1.12.0, installed in a separate environment.
• Current default main: 39e50ba, checked and reproduced on 2026-10-09.
• Originally reproduced and locally patched at 715b9a7 earlier the same day. The affected adapter is unchanged between these commits.

Tools Used

• Native / Native framework
• Opencompass backend
• VLMEvalKit backend
• RAGEval backend
• Perf / Model inference stress testing tool
• Arena / Arena mode

Executed Code or Instructions

Dependency installation requires internet once; the reproduction itself runs offline, blocks socket connections/name resolution, and needs no model, dataset download or API key.

python3.12 -m venv .venv
.venv/bin/python -m pip install evalscope==1.12.0
# Save the script below as repro.py:
.venv/bin/python repro.py
# This assertion check fails before the fix and passes after it:
.venv/bin/python repro.py --check

To reproduce on main instead, install the checkout editable into a separate environment and run the same script.

"""Offline replay of PubMedQA scoring with scripted model responses; no model calls."""
import json
import os
import sys

os.environ['HF_HUB_OFFLINE'] = '1'
os.environ['TRANSFORMERS_OFFLINE'] = '1'
os.environ['LITELLM_LOCAL_MODEL_COST_MAP'] = 'True'


def deny_network(event, args):
    if event in ('socket.connect', 'socket.getaddrinfo'):
        raise RuntimeError('This reproduction forbids network access')


sys.addaudithook(deny_network)

from evalscope.api.evaluator import TaskState
from evalscope.api.model import ModelOutput
from evalscope.api.registry import get_benchmark
from evalscope.config import TaskConfig


def replay(predictions):
    adapter = get_benchmark('pubmedqa', TaskConfig(datasets=['pubmedqa']))
    scores = []
    for index, (label, prediction) in enumerate(predictions):
        sample = adapter.record_to_sample({
            'context': 'A study abstract supplied by the dataset.',
            'question': 'Does the study support the stated hypothesis?',
            'answer': label.lower(),
            'reasoning': '',
        })
        sample.id = index
        state = TaskState(model='scripted-offline', sample=sample,
                          output=ModelOutput.from_content('scripted-offline', prediction), completed=True)
        scores.append(adapter.calculate_metrics(state))
    metrics = {('f1' if item.metric_name == 'f1_score' else item.metric_name): item.score
               for item in adapter.aggregate_scores(scores)}
    assert all(score.score.status.value == 'success' for score in scores)
    return metrics


def main():
    labels = ['YES', 'NO', 'MAYBE']
    # Same 30 cases. A: 8 correct and 2 abstentions per class.
    # B: 9 correct and 1 wrong label per class, cyclically distributed.
    model_a = [(label, label if i < 8 else 'Insufficient evidence.')
               for label in labels for i in range(10)]
    model_b = [(label, label if i < 9 else labels[(j + 1) % 3])
               for j, label in enumerate(labels) for i in range(10)]
    output = {'cases_per_model': 30, 'A': replay(model_a), 'B': replay(model_b),
              'expected_f1': {'A': 8 / 9, 'B': 0.9},
              'expected_recall': {'A': 0.8, 'B': 0.9}}
    print(json.dumps(output, indent=2))
    if '--check' in sys.argv:
        for name in ['A', 'B']:
            assert abs(output[name]['f1'] - output['expected_f1'][name]) < 1e-12
            assert abs(output[name]['recall'] - output['expected_recall'][name]) < 1e-12


if __name__ == '__main__':
    main()

Error Log

No scoring error or warning is emitted. Both the released package and current main produce these values (abridged from the script's JSON output):

{
  "A": {"accuracy": 0.8, "precision": 1.0, "recall": 1.0, "f1": 1.0},
  "B": {"accuracy": 0.9, "precision": 0.9, "recall": 0.9, "f1": 0.9},
  "expected_f1": {"A": 0.8888888888888888, "B": 0.9},
  "expected_recall": {"A": 0.8, "B": 0.9}
}

The --check invocation fails its F1 assertion for A.

Running Environment

• Operating System: macOS, Apple Silicon
• Python Version: 3.12.13
• No model endpoint, API key or judge configured.

Additional Information

Cause and proposed fix

In pubmed_qa_adapter.py, lines 123–133, an unrecognized prediction becomes None and is not inserted into the confusion matrix. Lines 151–156 then use the confusion-matrix row sum as the actual-positive count. This excludes precisely the unanswered cases that should count as false negatives.

Count ground-truth class support independently of whether a prediction was recognized:

             correct_count = 0
+            actual_counts = dict.fromkeys(confusion_matrix, 0)
 
             for ss in scores:
                 gt = ss.sample_metadata['answer'].strip().upper()
+                if gt in actual_counts:
+                    actual_counts[gt] += 1
@@
-                # Calculate actual positives (row sum)
-                act_pos = sum(confusion_matrix[cls][pred_cls] for pred_cls in classes)
+                # Unrecognized answers remain false negatives for their true class.
+                act_pos = actual_counts[cls]

The local patch also sets PubMedQA's evaluation_version to v1.1, following the repository's evaluation-semantics versioning requirement. It preserves the existing answer parser and three-class macro averaging.

Local regression validation

Nine cases in tests/benchmark/test_pubmedqa_aggregation.py exercise the normal scoring path: empty responses, two ordinary abstention phrases, the ranking example, uneven per-class support, and all-correct/all-wrong/all-abstaining/empty collections. Before the patch: 5 failed, 4 passed. After: 9 passed. Together with the 12 existing evaluation-version tests: 21 passed from the repository root. The reproduction after the patch gives A recall 0.8/F1 0.888888888888889 and leaves B at 0.9. Full repository tests and submission-wide checks have not been claimed.

Related reports checked

#1123 concerns benchmark availability; #919/#923 concern benchmark additions. #1575 concerns shared LLM-judge failure handling, whereas this reproduction uses successful rule-based scoring with no judge. The related fixes #1649, #1702 and #1816 do not change PubMedQA's recall denominator.

Disclosure: I maintain Driftproof, an evaluation tool. Found with AI-assisted code review and confirmed by the reproduction above.