The State of Peer Review in Empirical Software Engineering: A Community Survey on Review Load, Quality, and GenAI Use

2026-06-03

AI Summary

Peer review in empirical software engineering is increasingly strained by excessive reviewer workload, declining review quality, and emerging challenges posed by generative AI (GenAI). This study presents the first systematic survey within this community, gathering responses from 120 practitioners via a questionnaire and employing a mixed-methods approach that integrates quantitative and qualitative analyses. The research investigates perceived reviewing burdens, evaluations of review quality, practical applications of large language models (LLMs) in the peer review process, and associated ethical risks. Findings reveal widespread concerns about high workload and review quality, identify common usage patterns and potential pitfalls of GenAI in reviewing, and distill a set of community-driven recommendations for improving the peer review system—providing an empirical foundation for future, evidence-based reforms.

Abstract

The scientific peer review system has been slowly deteriorating over the last years, and not just within empirical software engineering (ESE) research. Increased submission numbers, high workload, and the rise of generative AI use with all its associated issues have made many cracks in the system more visible. To get a better understanding of the current state of peer review in the ESE community, we conducted a questionnaire survey, which accumulated 120 responses. We report on (i) the perceived review load of community members, (ii) review quality perception as well as frequent challenges for and issues with reviews, (iii) the use of LLM-based tools in the reviewing process, and (iv) the community's suggestions for improving the peer review system. We hope that these community opinions can facilitate more evidence-based discussions about how people want to see the review system change for the better.

Problem

• peer review
• review load
• review quality
• generative AI
• empirical software engineering

Innovation

• peer review
• empirical software engineering
• generative AI
• LLM-based tools
• review quality