Repository navigation
Feature request: add more MCQA benchmarks #919
Description
Hi! evalscope is amazingly good, I'm honestly so obsessed with adding benchmarks lately.
I noticed that there are many multiple-choice QA (MCQA) datasets that are widely used in literature but haven’t yet been added to evalscope. Some examples include:
- Social/Commonsense Reasoning: SiQA, PiQA, CommonsenseQA, LogiQA
- Science/Math/Medical QA: QASC, PubMedQA, MRI-MCQA, MathQA
- Hallucination: HaluEval
We've run these datasets ourselves and confirmed that the implementations are correct. Below is a quick summary table of results (subset of benchmarks I mentioned):
| Dataset | Llama3-8B | Llama3.1-8B | Qwen2.5-3B | Qwen2.5-7B | Qwen2.5-14B |
|---|---|---|---|---|---|
| SiQA | 0.7185 | 0.7236 | 0.7298 | 0.7805 | 0.7856 |
| PiQA | 0.8063 | 0.8270 | 0.7715 | 0.8705 | 0.8917 |
| CommonsenseQA | 0.7453 | 0.7420 | 0.7559 | 0.8206 | 0.8206 |
| LogiQA | 0.4363 | 0.4347 | 0.4347 | 0.5376 | 0.6160 |
| QASC | 0.7624 | 0.7959 | 0.7203 | 0.7959 | 0.8294 |
| PubMedQA | 0.6300 | 0.7090 | 0.5840 | 0.5360 | 0.4480 |
| MRI-MCQA | 0.5258 | 0.5488 | 0.4938 | 0.5595 | 0.6004 |
| MathQA | 0.5528 | 0.6945 | 0.7725 | 0.8328 | 0.8516 |
| HaluEval-D | 0.8210 | 0.8022 | 0.5219 | 0.9087 | 0.9404 |
| HaluEval-Q | 0.7898 | 0.6833 | 0.4914 | 0.6733 | 0.8839 |
| HaluEval-S | 0.7174 | 0.6552 | 0.4545 | 0.5875 | 0.8490 |
Would you be interested in adding these benchmarks to evalscope? I think they’d really strengthen coverage for reasoning and domain-specific evaluation. If positive, I'll submit a PR.
Warm regards,
Yang
Activity
Yunnglin commentedon Oct 26, 2025
Thank you for this excellent proposal! I'm very interested in adding these MCQA benchmarks to evalscope.
The datasets you've identified would significantly strengthen our coverage for reasoning and domain-specific evaluation. I really appreciate that you've already validated the implementations and provided the comprehensive benchmark results.
Please go ahead and submit a PR - we'd love to have these benchmarks included!
Thanks again for your contribution to the project.
penguinwang96825 commentedon Oct 27, 2025
Since PR #923 has resolved this issue, I'll close it for now. Thanks for all the support!