[mavericksea-ai]

Problem and change

PubMedQA drops unrecognized answers from its confusion matrix, then uses the matrix row sums as the recall denominators. This silently inflates macro recall/F1. In the reproduction in #1827, a model answering 24 of 30 cases correctly and abstaining on six receives F1 1.0, incorrectly ranking ahead of a model scoring 0.9.

Count ground-truth class support independently of recognized predictions, so abstentions remain false negatives. The first model's F1 becomes 0.888889 and recall becomes 0.8; accuracy and the other model's scores are unchanged. Preserve the existing three-class macro averaging and answer parsing. Bump PubMedQA's evaluation version to v1.1 for the changed scoring semantics.

Fixes #1827.

Validation

• Nine regression cases cover empty/unrecognized replies, uneven class support, the ranking reversal, and boundary controls. Before the fix: 5 fail / 4 pass; afterward: 9 pass.
• Regression cases plus existing evaluation-version tests: 21 passed.
• make lint: passed.
• pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings: passed using the mock model (no .env file/API key).
• No model API calls. The issue includes a standalone offline reproduction confirmed on PyPI 1.12.0 and current main.

[Yunnglin]

LGTM — verified locally that the recall-denominator fix is correct (unrecognized answers now stay false negatives, precision untouched), the 9 regression tests pass, and reverting the one-line act_pos change reproduces the reported 5 fail / 4 pass; thanks for the thorough reproduction and the v1.1 bump!