Identifying Aspects in Peer Reviews

2025-04-09

AI Summary

Prior work lacks a well-defined, data-driven approach to identifying fine-grained review dimensions in peer review. Method: This paper proposes the first fully text-based, data-driven framework for discovering review dimensions directly from real review texts—eschewing reliance on predefined review forms—and adopts a bottom-up paradigm integrating language modeling, unsupervised clustering, topic modeling, and semantic similarity analysis, augmented by human-in-the-loop annotation for dimension induction and validation. Contributions/Results: (1) We release the first high-quality, reusable dataset of expert-annotated review dimensions; (2) we establish an operational, empirically grounded definition system for “review dimensions”; (3) we empirically demonstrate that dimension selection critically impacts downstream tasks—e.g., detecting LLM-generated reviews—with robust cross-conference consistency and alignment with community values, achieving significant gains in detection accuracy.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Peer review is central to academic publishing, but the growing volume of submissions is straining the process. This motivates the development of computational approaches to support peer review. While each review is tailored to a specific paper, reviewers often make assessments according to certain aspects such as Novelty, which reflect the values of the research community. This alignment creates opportunities for standardizing the reviewing process, improving quality control, and enabling computational support. While prior work has demonstrated the potential of aspect analysis for peer review assistance, the notion of aspect remains poorly formalized. Existing approaches often derive aspect sets from review forms and guidelines of major NLP venues, yet data-driven methods for aspect identification are largely underexplored. To address this gap, our work takes a bottom-up approach: we propose an operational definition of aspect and develop a data-driven schema for deriving fine-grained aspects from a corpus of peer reviews. We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis. We further show how the choice of aspects can impact downstream applications, such as LLM-generated review detection. Our results lay a foundation for a principled and data-driven investigation of review aspects, and pave the path for new applications of NLP to support peer review.

Problem

• Formalizing aspects in peer reviews for standardization
• Developing data-driven methods for aspect identification
• Exploring impact of aspects on NLP applications

Innovation

• Data-driven schema for aspect identification
• Operational definition of review aspects
• Dataset for community-level review analysis