Problem
Research questions and friction points this paper is trying to address.
code review
AI coding agents
developer response
comment resolution
software engineering
"Go Home Copilot, You're Drunk": Understanding Developer Responses to Agent-Generated Code Review Comments
🤖 AI Summary
This study addresses the limited understanding of how developers respond to code review comments generated by AI coding agents, a gap that hinders the effective integration of AI-assisted reviewing. Through a large-scale empirical analysis of 54,791 AI-generated review comments across 342 Python repositories—produced by prominent agents including Copilot, Cursor, Codex, Devin, and Claude—the authors combine GitHub data mining, quantitative statistics, and open card sorting to uncover real-world developer response patterns. The findings reveal that Copilot accounts for 72.9% of resolved comments; core developers primarily address design-related feedback, while peripheral contributors focus on functional defects; unresolved comments often stem from incorrect suggestions or deliberate design choices; and inline code suggestions significantly increase adoption rates. The study further identifies ten root causes underlying unresolved feedback.
📝 Abstract
Code review is a critical quality assurance practice in software engineering development, and AI coding agents are increasingly generating review comments on pull requests. However, little is known about how developers actually respond to such agent-generated feedback. In this paper, we present the first large-scale empirical study on the resolution of agent-generated code review comments. We analyze $54{,}791$ comments generated by five widely used coding agents (i.e., Copilot, Cursor, Codex, Devin, and Claude) across $342$ Python repositories on GitHub. We examine (1) resolution rates across agents and comment types, (2) the role of developer experience, and (3) characteristics that influence comment usefulness. Our results show that resolution rate varies considerably across agents, with Copilot accounting for the majority of resolved comments (72.9\%). Core developers resolve the majority of agent-generated feedback, particularly for \textit{design} and \textit{evolvability}-related comments, while peripheral developers are more involved in resolving \textit{functional defect} comments. Through open card sorting of 470 unresolved comment discussions, we identify \textit{ten} discussion patterns explaining why comments remain unresolved, with \textit{incorrect suggestions} and \textit{intentional design decisions} being the most prevalent. Finally, our analysis reveals that the presence of an inline \textit{code suggestion} is the strongest predictor of comment resolution, while lengthy and complex comments are less likely to be acted upon. Our findings provide insights for improving AI-generated code review feedback and its integration into development workflows.
Innovation
Methods, ideas, or system contributions that make the work stand out.
AI code review
developer response
empirical study
code suggestion
resolution rate
🔎 Similar Papers
No similar papers found.
