You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This repository collects papers, datasets, benchmarks, and tools for AI-assisted peer review. It follows the survey taxonomy: peer review generation, after-review tasks, and benchmark perspectives.
Peer review is a multi-stage workflow involving review generation, rebuttal, discussion, meta-review, final decision, and manuscript revision. This list is intended to support research on AI-assisted peer review workflows, not to replace human reviewers.
✨ Highlights
Organized by the survey taxonomy: Generation, After Peer Review, and Evaluation.
Covers fine-tuning, agent-based systems, reinforcement learning, and generation enhancement for peer review generation.
Includes post-review tasks such as rebuttal generation, meta-review generation, and paper revision from reviews.
Separates evaluation into human-centric, reference-based, LLM-based, and aspect-oriented paradigms.
Tracks representative datasets, benchmarks, and data collection tools.
[PeerRead] A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications - NAACL 2018
[NLpeer] NLpeer: A Unified Resource for the Computational Study of Peer Review - ACL 2023
[MOPRD] MOPRD: A Multidisciplinary Open Peer Review Dataset - NCA 2023
Early Review-related Methods
[ReviewRobot] ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis - INLG 2020
[ASAP-Review] Can We Automate Scientific Reviewing? - EMNLP 2021
[PEERAssist] PEERAssist: Leveraging on Paper-Review Interactions to Predict Peer Review Decisions - ICADL 2021
[Peer-Review Score Prediction] Exploiting Labeled and Unlabeled Data via Transformer Fine-tuning for Peer-Review Score Prediction - EMNLP Findings 2022
Fine-tuning Methods
[ReviewMT] Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions - arXiv 2024
[OpenReviewer] OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews - NAACL Demo 2025
[ReviewMT] Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions - arXiv 2024
[MATEval] MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation - arXiv 2024
[PaperEval] PaperEval: A Universal, Quantitative, and Explainable Paper Evaluation Method Powered by a Multi-Agent System - Information Processing & Management 2025
Reinforcement Learning Methods
[CycleResearcher] CycleResearcher: Improving Automated Research via Automated Review - ICLR 2025
[REMOR] REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning - arXiv 2025
[ReviewRL] ReviewRL: Towards Automated Scientific Review with RL - EMNLP 2025
[REM-CTX] REM-CTX: Automated Peer Review via Reinforcement Learning with Auxiliary Context - arXiv 2026
Review Generation Enhancement
External Knowledge and Retrieval
[ReviewRobot] ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis - INLG 2020
[ReviewMT] Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions - arXiv 2024
[Re²] Re²: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions - arXiv 2025
[DRPG] DRPG (Decompose, Retrieve, Plan, Generate): An Agentic Framework for Academic Rebuttal - arXiv 2026
[Paper2Rebuttal] Paper2Rebuttal: A Multi-Agent Framework for Transparent Author Response Assistance - arXiv 2026
[Author-in-the-loop] Author-in-the-loop Response Generation and Evaluation: Integrating Author Expertise and Intent in Responses to Peer Review - ACL 2026
Meta-review
[MRED] MReD: A Meta-Review Dataset for Structure-Controllable Text Generation - ACL Findings 2022
[PEERSUM] Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation - EMNLP Findings 2023
[ORSUM] Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluation - IJCAI 2024 Workshop
[Automated Meta-review Pipeline] Towards Automated Meta-review Generation via an NLP/ML Pipeline in Different Stages of the Scholarly Peer Review Process - International Journal on Digital Libraries 2024
[PeerArg] PeerArg: Argumentative Peer Review with LLMs - NeLaMKRR 2024
Paper Revision from Reviews
[ARXIVEDITS] arXivEdits: Understanding the Human Revision Process in Scientific Writing - EMNLP 2022
[ARIES] ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews - ACL 2024
[CASIMIR] CASIMIR: A Corpus of Scientific Articles enhanced with Multiple Author-Integrated Revisions - LREC-COLING 2024
📊 Evaluation and Benchmarks
Human-Centric Evaluation
[GPT-4 Pilot Study] GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Study - arXiv 2023
[Useful Feedback] Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis - NEJM AI 2023
[Reviewer Fatigue or Bias] Fighting Reviewer Fatigue or Amplifying Bias? Considerations and Recommendations for Use of ChatGPT and Other Large Language Models in Scholarly Peer Review - Research Integrity and Peer Review 2023
[Reviewer Arena] AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviews - arXiv 2024
[Medical Peer Review with LLMs] The Role of Large Language Models in the Peer-review Process: Opportunities and Challenges for Medical Journal Reviewers and Editors - J Educ Eval Health Prof 2025
Reference-Based Evaluation
[ReviewerGPT] ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing - arXiv 2023
[RR-MCQ] Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks - LREC-COLING 2024
[SEA] Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis - EMNLP 2024
[Useful Feedback] Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis - NEJM AI 2023
LLM-Based Evaluation
[Critical Problems] Reviewing Scientific Papers for Critical Problems With Reasoning LLMs: Baseline Approaches and Automatic Evaluation - NeurIPS 2025 Workshop
[PiCO] PiCO: Peer Review in LLMs based on the Consistency Optimization - ICLR 2025
[Auto-PRE] Auto-PRE: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation - AAAI 2026
Aspect-Oriented Evaluation
[Focus-Level Framework] Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews - EMNLP 2025
If you find this repository useful, please consider citing our paper:
@misc{wu2026aigoodpeerreviewer,
title={Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future},
author={Sihong Wu and Owen Jiang and Yilun Zhao and Tiansheng Hu and Yiling Ma and Kaiyan Zhang and Manasi Patwardhan and Arman Cohan},
year={2026},
eprint={2604.27924},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2604.27924},
}
