Test Coverage Analysis of Agentic Pull Requests

2026-07-20

AI Summary

This study addresses the pervasive issue of insufficient test coverage in pull requests generated by AI coding agents, particularly highlighting severe gaps in testing error-handling code, which undermines code quality and regression prevention. Analyzing 4,882 AI-generated Java and Python pull requests, the work combines static analysis with dynamic test coverage tracking to quantitatively assess line-level coverage of modified code. It finds that only 49.6% of pull requests include any test changes, and existing tests cover merely 61.5% of Java and 27.0% of Python modified lines. Newly added tests by agents rarely improve coverage, and error-handling constructs exhibit alarmingly high omission rates of 81%–86%. To address these blind spots, the paper proposes a coverage-driven feedback mechanism tailored for AI coding agents.

Abstract

AI coding agents increasingly submit complete pull requests (PRs) with minimal human intervention, shifting software development from AI-assisted to autonomous workflows. As these agents become more prevalent, ensuring the code they generate is adequately tested, by existing tests or by tests the agents write, is critical to preventing regressions, yet little is known about testing in agentic PRs. To address this gap, we analyze 4882 agent-generated PRs from the AIDev dataset (532 Java and 4350 Python PRs) produced by five coding agents. We study (i) how often agents include test changes and (ii) how well covered are code changes by existing and agent-written tests. Agents include test changes in only 49.6% of PRs that change code under test files. Existing tests provide an incomplete safety net: they cover 61.5% of agents' changed executable lines in Java and only 27.0% in Python, where 64.8% of PRs have no changed line executed by any existing test. Agent-written tests improve coverage over existing tests, but only in a minority of PRs: 35.9% of Java and 22.5% of Python Code + Tests PRs show a coverage gain. Across both languages, error-handling constructs (e.g., try and catch blocks) are the most consistently under-tested, with miss rates reaching 86.0% in Java and 81.0% in Python. These findings motivate coverage-aware development practices, coverage feedback loops for coding agents, and evaluation benchmarks that measure test quality to better help agents reliably test their own code.

Problem

• test coverage
• AI coding agents
• pull requests
• software testing
• autonomous code generation

Innovation

• test coverage
• AI coding agents
• pull request analysis
• autonomous software development
• coverage feedback