Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI

2024-06-20

Scientometrics

AI Summary

This study investigates the alignment between reviewers’ confidence scores and textual content in AI top-conference peer reviews, addressing a gap in fine-grained alignment analysis. We propose the first multi-granular (word-, sentence-, and aspect-level) quantitative alignment framework, integrating deep learning–based aspect extraction and fuzzy sentence detection with NLP features—including sentiment polarity, hedge word/sentence frequency, and comment length—and multi-level statistical tests (correlation, regression, significance). Results show strong alignment between confidence scores and review text across all granularities. Counterintuitively, higher confidence scores significantly predict paper rejection (p < 0.01), supporting reviewer expertise and the credibility of the review process. Our core contribution is the first cross-granular confidence–text alignment paradigm, demonstrating that confidence scores serve not merely as ordinal indicators but as interpretable, high-fidelity proxies for review quality.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Knowledge Representation and Reasoning: Preferences

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Peer review is vital in academia for evaluating research quality. Top AI conferences use reviewer confidence scores to ensure review reliability, but existing studies lack fine-grained analysis of text-score consistency, potentially missing key details. This work assesses consistency at word, sentence, and aspect levels using deep learning and NLP conference review data. We employ deep learning to detect hedge sentences and aspects, then analyze report length, hedge word/sentence frequency, aspect mentions, and sentiment to evaluate text-score alignment. Correlation, significance, and regression tests examine confidence scores' impact on paper outcomes. Results show high text-score consistency across all levels, with regression revealing higher confidence scores correlate with paper rejection, validating expert assessments and peer review fairness.

Problem

• Assess consistency between reviewer confidence scores and review content
• Analyze text-score alignment at word, sentence, and aspect levels
• Examine impact of confidence scores on paper acceptance outcomes

Innovation

• Deep learning detects hedge sentences and aspects
• Analyzes text-score alignment via sentiment and frequency
• Regression tests link confidence scores to outcomes