When AI Agents Touch CI/CD Configurations: Frequency and Success

2026-01-24

AI Summary

This study presents the first large-scale analysis of AI agents’ modifications to CI/CD configurations, examining 8,031 pull requests authored by AI agents—such as Copilot, Codex, and Devin—across 1,605 GitHub repositories, with a focus on YAML-based CI/CD workflow files. The findings reveal that CI/CD-related changes constitute 3.25% of all AI-generated modifications, with 96.77% targeting GitHub Actions. While the overall build success rate for AI-authored changes (75.59%) is comparable to that of human-authored ones (74.87%), their merge rate is slightly lower. Notably, Copilot demonstrates a significantly higher merge rate (+15.63 percentage points) in CI/CD tasks, suggesting emerging specialization among AI agents in automated DevOps workflows and highlighting behavioral differences across agent types in infrastructure automation contexts.

Technology Category

• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent/AI Theories and Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

AI agents are increasingly used in software development, yet their interaction with CI/CD configurations is not well studied. We analyze 8,031 agentic pull requests (PRs) from 1,605 GitHub repositories where AI agents touch YAML configurations. CI/CD configuration files account for 3.25% of agent changes, varying by agent (Devin: 4.83%, Codex: 2.01%, p<0.001). When agents modify CI/CD, 96.77% target GitHub Actions. Agentic PRs with CI/CD changes merge slightly less often than others (67.77% vs. 71.80%), except for Copilot, whose CI/CD changes merge 15.63 percentage points more often. Across 99,930 workflow runs, build success rates are comparable for CI/CD and non-CI/CD changes (75.59% vs. 74.87%), though three agents show significantly higher success when modifying CI/CD. These results show that AI agents rarely modify CI/CD and focus mostly on GitHub Actions, yet their configuration changes are as reliable as regular code. Copilot's strong CI/CD performance despite lower acceptance suggests emerging configuration specialization, with implications for agent training and DevOps automation.

Problem

• AI agents
• CI/CD configurations
• GitHub Actions
• pull requests
• DevOps automation

Innovation

• AI agents
• CI/CD configurations
• GitHub Actions
• DevOps automation
• pull request success