Beyond Bug Fixes: An Empirical Investigation of Post-Merge Code Quality Issues in Agent-Generated Pull Requests

2026-01-27

AI Summary

This study addresses the lack of systematic evaluation of code quality in real-world projects for pull requests (PRs) generated by AI coding agents that have been successfully merged. Analyzing 1,210 merged AI-generated Python fix PRs, the authors employ differential static analysis via SonarQube and normalize findings against multiple dimensions, including code churn, to assess newly introduced issues across five AI agents. Their findings reveal that successful merging does not guarantee code quality: code smells dominate and are often high-severity, while bugs—though less frequent—are frequently critical. After normalization by PR size, the disparity in issue density across agents markedly diminishes, suggesting that problem prevalence stems primarily from PR scale rather than inherent agent capability. This work underscores significant quality risks in AI-generated code and calls for standardized quality assurance mechanisms.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Intelligent Robots: Embodied AI

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The increasing adoption of AI coding agents has increased the number of agent-generated pull requests (PRs) merged with little or no human intervention. Although such PRs promise productivity gains, their post-merge code quality remains underexplored, as prior work has largely relied on benchmarks and controlled tasks rather than large-scale post-merge analyses. To address this gap, we analyze 1,210 merged agent-generated bug-fix PRs from Python repositories in the AIDev dataset. Using SonarQube, we perform a differential analysis between base and merged commits to identify code quality issues newly introduced by PR changes. We examine issue frequency, density, severity, and rule-level prevalence across five agents. Our results show that apparent differences in raw issue counts across agents largely disappear after normalizing by code churn, indicating that higher issue counts are primarily driven by larger PRs. Across all agents, code smells dominate, particularly at critical and major severities, while bugs are less frequent but often severe. Overall, our findings show that merge success does not reliably reflect post-merge code quality, highlighting the need for systematic quality checks for agent-generated bug-fix PRs.

Problem

• code quality
• AI coding agents
• pull requests
• post-merge analysis
• code smells

Innovation

• AI coding agents
• post-merge code quality
• differential analysis
• code smells
• pull request