On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub

2025-09-18

AI Summary

Despite growing interest in large language model (LLM)-driven autonomous coding agents, their practical utility in real-world open-source development remains empirically underexplored. Method: This study conducts the first large-scale empirical evaluation of Claude Code—an LLM-based autonomous coding agent—in authentic open-source settings, analyzing 567 pull requests (PRs) it generated across 157 GitHub projects. We systematically assess PR acceptance rates, revision requirements, and human–agent collaboration patterns. Contribution/Results: 83.8% of PRs were merged; 54.9% were accepted without modification, while the remainder underwent lightweight developer refinements before integration—consistently improving code quality and adherence to project-specific conventions. The analysis reveals effective human–AI division of labor in refactoring, documentation generation, and test creation. These findings provide critical empirical evidence and a practical framework for integrating AI coding agents into industrial software development workflows.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) are increasingly being integrated into software development processes. The ability to generate code and submit pull requests with minimal human intervention, through the use of autonomous AI agents, is poised to become a standard practice. However, little is known about the practical usefulness of these pull requests and the extent to which their contributions are accepted in real-world projects. In this paper, we empirically study 567 GitHub pull requests (PRs) generated using Claude Code, an agentic coding tool, across 157 diverse open-source projects. Our analysis reveals that developers tend to rely on agents for tasks such as refactoring, documentation, and testing. The results indicate that 83.8% of these agent-assisted PRs are eventually accepted and merged by project maintainers, with 54.9% of the merged PRs are integrated without further modification. The remaining 45.1% require additional changes benefit from human revisions, especially for bug fixes, documentation, and adherence to project-specific standards. These findings suggest that while agent-assisted PRs are largely acceptable, they still benefit from human oversight and refinement.

Problem

• Evaluating acceptance rate of AI-generated pull requests
• Assessing practical usefulness of autonomous coding agents
• Analyzing human intervention needs in agent-assisted contributions

Innovation

• Agentic coding for automated pull requests
• Empirical analysis of AI-generated contributions
• Human oversight enhances AI integration success