How AI Coding Agents Communicate: A Study of Pull Request Description Characteristics and Human Review Responses

2026-02-19

AI Summary

This study investigates how the descriptive characteristics of pull requests (PRs) submitted by AI coding agents on GitHub influence human reviewers’ response behaviors. Leveraging the AIDev dataset, we conduct an empirical analysis of PRs from five prominent AI agents, integrating natural language processing and software engineering log mining to quantify associations between textual features of PR descriptions and human review activity, response latency, sentiment, and merge outcomes. Our work is the first to systematically reveal significant stylistic differences among AI agents in PR presentation and their impact on human-AI collaborative code review. The findings demonstrate that description style substantially affects reviewer engagement, response speed, and merge success rates, with distinct performance patterns across agents in human interaction metrics—highlighting the critical role of communicative expression in AI-assisted programming.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The rapid adoption of large language models has led to the emergence of AI coding agents that autonomously create pull requests on GitHub. However, how these agents differ in their pull request description characteristics, and how human reviewers respond to them, remains underexplored. In this study, we conduct an empirical analysis of pull requests created by five AI coding agents using the AIDev dataset. We analyze agent differences in pull request description characteristics, including structural features, and examine human reviewer response in terms of review activity, response timing, sentiment, and merge outcomes. We find that AI coding agents exhibit distinct PR description styles, which are associated with differences in reviewer engagement, response time, and merge outcomes. We observe notable variation across agents in both reviewer interaction metrics and merge rates. These findings highlight the role of pull request presentation and reviewer interaction dynamics in human-AI collaborative software development.

Problem

• AI coding agents
• pull request description
• human review response
• GitHub
• human-AI collaboration

Innovation

• AI coding agents
• pull request description
• human-AI collaboration
• code review
• empirical study