Re$^2$: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions

2025-05-12

AI Summary

The surge in AI paper submissions has strained peer review resources, degraded review quality, and created a critical shortage of high-quality data to support author self-assessment. Existing review datasets suffer from limited diversity, low fidelity (often using revised rather than original manuscripts), and an inability to model the dynamic review-rebuttal interaction. Method: We introduce the first large-scale, high-fidelity, full-cycle peer review dataset featuring 19K+ submissions, 70K+ reviews, and 53K+ rebuttals across 24 conferences and 21 workshops. It establishes a novel paradigm—structured multi-turn dialogue grounded in original manuscripts, with unified human annotation across submission → review → rebuttal → discussion stages. Data reliability is ensured via OpenReview-sourced real-world data, multi-tier manual verification, and automated consistency checking. Contribution/Results: This dataset substantially improves LLM performance in review summarization, critique generation, and rebuttal writing—boosting accuracy and consistency—and enables the development of actionable pre-submission self-evaluation tools.

Technology Category

• Humans and AI: Crowd Sourcing and Human Computation
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

Peer review is a critical component of scientific progress in the fields like AI, but the rapid increase in submission volume has strained the reviewing system, which inevitably leads to reviewer shortages and declines review quality. Besides the growing research popularity, another key factor in this overload is the repeated resubmission of substandard manuscripts, largely due to the lack of effective tools for authors to self-evaluate their work before submission. Large Language Models (LLMs) show great promise in assisting both authors and reviewers, and their performance is fundamentally limited by the quality of the peer review data. However, existing peer review datasets face three major limitations: (1) limited data diversity, (2) inconsistent and low-quality data due to the use of revised rather than initial submissions, and (3) insufficient support for tasks involving rebuttal and reviewer-author interactions. To address these challenges, we introduce the largest consistency-ensured peer review and rebuttal dataset named Re^2, which comprises 19,926 initial submissions, 70,668 review comments, and 53,818 rebuttals from 24 conferences and 21 workshops on OpenReview. Moreover, the rebuttal and discussion stage is framed as a multi-turn conversation paradigm to support both traditional static review tasks and dynamic interactive LLM assistants, providing more practical guidance for authors to refine their manuscripts and helping alleviate the growing review burden. Our data and code are available in https://anonymous.4open.science/r/ReviewBench_anon/.

Problem

• Addressing reviewer shortages and declining review quality in AI peer review systems
• Overcoming limitations of existing peer review datasets: diversity, consistency, and interaction support
• Providing a large, consistency-ensured dataset for multi-turn rebuttal discussions and LLM assistance

Innovation

• Largest consistency-ensured peer review dataset
• Multi-turn conversation paradigm for rebuttals
• Supports dynamic interactive LLM assistants