Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards

2025-05-08

AI Summary

Surging submission volumes at top-tier AI conferences (exceeding 10,000 papers per cycle) have degraded review quality and weakened reviewer accountability. Method: We propose a bidirectional peer-review mechanism wherein authors anonymously evaluate review quality, and reviewers earn cumulative academic credit tied to evaluation outcomes. Contribution/Results: Our work introduces (i) the first closed-loop feedback architecture linking authors and reviewers; (ii) a two-stage anonymous evaluation protocol designed to prevent retaliation; and (iii) the first credit-based measurement and incentive framework tailored for AI conference reviewing. Through formal mechanism design and game-theoretic modeling, our approach simultaneously enhances review fairness and participant engagement. It provides a practical institutional pathway toward a sustainable, high-fidelity AI peer-review ecosystem—facilitating community consensus and systemic reform.

Technology Category

• Multiagent Systems: Mechanism Design
• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Humans and AI: Planning and Decision Support for Human-Machine Teams

Application Category

• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The peer review process in major artificial intelligence (AI) conferences faces unprecedented challenges with the surge of paper submissions (exceeding 10,000 submissions per venue), accompanied by growing concerns over review quality and reviewer responsibility. This position paper argues for the need to transform the traditional one-way review system into a bi-directional feedback loop where authors evaluate review quality and reviewers earn formal accreditation, creating an accountability framework that promotes a sustainable, high-quality peer review system. The current review system can be viewed as an interaction between three parties: the authors, reviewers, and system (i.e., conference), where we posit that all three parties share responsibility for the current problems. However, issues with authors can only be addressed through policy enforcement and detection tools, and ethical concerns can only be corrected through self-reflection. As such, this paper focuses on reforming reviewer accountability with systematic rewards through two key mechanisms: (1) a two-stage bi-directional review system that allows authors to evaluate reviews while minimizing retaliatory behavior, (2)a systematic reviewer reward system that incentivizes quality reviewing. We ask for the community's strong interest in these problems and the reforms that are needed to enhance the peer review process.

Problem

• Addressing AI conference peer review crisis with bi-directional feedback
• Reforming reviewer accountability through systematic rewards
• Enhancing review quality via author feedback and reviewer incentives

Innovation

• Two-stage bi-directional review system
• Systematic reviewer reward system
• Author feedback minimizes retaliatory behavior