Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Dataset

2026-06-11

AI Summary

This study addresses the high rejection rate of code fixes generated by AI programming agents, which leads to significant waste of human and computational resources. Drawing on the AIDev dataset, the authors conduct a mixed-methods analysis—combining qualitative content analysis with quantitative statistics—of 306 unmerged pull requests produced by prominent AI agents such as Copilot, Devin, Cursor, and Claude. They systematically identify and categorize 14 distinct reasons for rejection into four overarching themes: implementation errors, continuous integration (CI) failures, inherent capability limitations, and insufficient priority alignment. The findings reveal that 46.41% of AI-generated fixes are ultimately rejected, underscoring critical directions for improving human-AI collaboration: refining repair strategy guidance, enhancing constraint-aware prompting, and strengthening validation mechanisms. This work provides empirical grounding for advancing the reliability and collaborative efficacy of AI programming agents.

Abstract

AI coding agents are increasingly used to generate pull requests (PRs) that propose code fixes in software projects. From a first exploration of the AIDev dataset, we find that 46.41\% of the fixes proposed by the agents Copilot, Devin, Cursor, and Claude are rejected. This represents a significant amount of wasted resources that require human reviews, verifications, and running tests and validations for fixes that are merely discarded. Our goal in this paper is to understand the failure modes of AI-agents, an understanding that is crucial for better integrating AI-agents as efficient teammates. In this paper, we conduct a qualitative study on a representative sample of 306 non-merged pull requests created or co-authored by the agents mentioned earlier, followed by a quantitative analysis of the reasons for rejection. Our qualitative findings identify 14 reasons divided into four high-level categories for rejecting AI-agent fixes. We observe that developers can reject fixes due to fixes whose implementation is incorrect (e.g., incomplete, wrong approach), fixes that do not pass the continuous integration (CI) pipelines and fail tests, fixes for which the agent is unable to perform the implementation (e.g., no code generated, sessions lost), and fixes whose priority is low. Our results shed light on the importance of better guiding the model at these levels: (1) proposing hints about the approach to follow for fixing an issue, (2) outlining constraints or limitations regarding the approaches that should not be taken, and (3) instructing the agent on how to validate the implementation through CI pipelines and without introducing a breaking change. Our results suggest the need for good prioritization of tasks so that generated fixes do not lead to wasted human review efforts or wasted agent resources (e.g., tokens, compute, or allowed number of requests).

Problem

• AI coding agents
• pull request rejection
• fix failure modes
• software maintenance
• code review

Innovation

• AI coding agents
• pull request rejection
• failure mode analysis
• continuous integration
• human-AI collaboration