What Do AI Agents Actually Change? An Empirical Taxonomy of Mutation Patterns in Performance-Improving Pull Requests

2026-07-06

AI Summary

This study addresses the lack of transparency in code-modification behaviors of current AI-powered coding agents during performance optimization. It presents the first empirical analysis of 1,254 code diffs from 216 AI-generated performance-optimization pull requests, systematically annotated using a dual-LLM cross-validation protocol based on an 18-category syntactic mutation taxonomy derived from Genetic Improvement (GI). The findings reveal a strong preference by AI agents for three mutation types: identifier renaming (37.0%), object creation (26.4%), and type changes (22.7%)—a stark contrast to traditional GI datasets, where over 84% of mutations involve no change. These results suggest that agent identity and optimization strategy can serve as effective priors for narrowing the search space of Search-Based Software Engineering (SBSE) operators.

Abstract

AI coding agents are black boxes: we cannot inspect how they generate code, but we can inspect what they change. This distinction matters for search-based software engineering (SBSE), where techniques such as genetic improvement (in the performance-optimisation application we study) depend on mutation operators that reflect how code is actually transformed. Fewer than 1% of the 33,596 agent PRs in AIDev-pop target performance, making each case a rare window into otherwise opaque agent behaviour. We classify 1,254 performance-relevant diff hunks from 216 of these PRs, spanning five agent systems, against the 18-category syntactic mutation taxonomy of Even-Mendoza et al. (2025) using a dual-LLM intersection pipeline. Three categories dominate: name modification (37.0%), object creation (26.4%), and type change (22.7%), a profile markedly different from prior GI corpora where no change accounted for 84%. Each agent's deployed system commits to a distinctive mutation vocabulary, and each performance strategy activates a largely disjoint category subset. Agent identity and target strategy are therefore informative priors that narrow the effective SBSE operator space. Replication package: https://github.com/5uper6rain/ssbse-challenge-2026

Problem

• AI coding agents
• mutation patterns
• performance optimization
• search-based software engineering
• genetic improvement

Innovation

• AI coding agents
• mutation patterns
• genetic improvement
• search-based software engineering
• empirical taxonomy