Can AI Review Improve Paper Drafting? An Empirical Study on 20 Computer Architecture Submissions

2026-05-31

AI Summary

This study investigates the potential of AI-generated peer reviews to assist in scientific paper writing under constraints of limited human reviewer availability. The authors develop AI-Paper-Review, an integrated web-based tool, and conduct a case study on 20 computer architecture papers to systematically quantify the overlap and divergence between AI and human reviewers in identifying issues. They introduce a novel comment clustering and ranking mechanism based on consensus and perceived importance. Experimental results demonstrate that the AI system effectively reproduces most concerns raised by human reviewers and uncovers certain critical flaws overlooked by them, highlighting its utility in early manuscript revision stages. However, the findings also indicate that current AI capabilities remain insufficient for formal peer review. The project’s code and data are publicly released.

Abstract

Research is advancing faster than ever with artificial intelligence (AI); and so are the corresponding research papers. The exploding volume of AI-generated papers have put a strain to peer review, leading to the usage of AI-generated review, potentially wide yet sneaky. However, relevant ethical concerns about confidentiality, quality, and fairness are raised and no consensus has been reached in the broad research community. We expect the debate to continue for a while, but in the meantime, we ask an alternative, practical question: \textit{can AI review improve paper drafting?} We study 20 computer architecture papers, with varying levels of submission lineage, to expose how well AI review aligns with human review, quantified by a set of metrics we define. To conduct the case study, we build a web UI-integrated tool, \emph{AI-Paper-Review}, that generates structured AI review of a draft paper, available at https://github.com/unarylab/ai-paper-review. This tool selects several AI reviewers from a diverse pool of AI reviewers and clusters and ranks their comments based on commonality and importance of review comments. It also allows to align AI comments with human comments to facilitate metric-based validation. The case study shows that AI review can cover a significant fraction of human-raised issues, but also raises issues missing in human review. This paper is not intended to encourage using AI for peer review at the current stage, but to study that (1) how AI review can improve paper drafting and (2) the potential and limitation of AI-based peer review. The release of the tool and the case study data is intended to instigate future research on this topic. Misuse for peer review would violate the ethics policies from major academic venues.

Problem

• AI review
• paper drafting
• peer review
• computer architecture
• AI-generated feedback

Innovation

• AI peer review
• paper drafting assistance
• structured review generation
• review comment clustering
• human-AI alignment