Exploringand Unleashing the Power of Large Language Models in CI/CD Configuration Translation

2025-11-03

AI Summary

This work addresses the challenge of automated configuration migration across CI platforms—particularly from Travis CI to GitHub Actions—where manual translation is error-prone and labor-intensive. We propose an LLM-based translation framework grounded in empirical analysis of 811 real-world migration cases, enabling the first quantitative characterization of configuration conversion effort. We introduce a four-category taxonomy of translation problems and identify recurring developer pain points. Methodologically, we design a composite prompting strategy integrating documentation-guided instruction, iterative refinement, and in-context learning to enhance LLM robustness and fidelity. Evaluation on GPT-4o shows our approach achieves 75.5% end-to-end build success rate—nearly tripling the performance of baseline prompting—while substantially reducing manual intervention. The framework provides a reproducible, quantitatively evaluable pathway for intelligent CI/CD configuration migration.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Continuous Integration (CI) is a cornerstone of modern collaborative software development, and numerous CI platforms are available. Differences in maintenance overhead, reliability, and integration depth with code-hosting platforms make migration between CI platforms a common practice. A central step in migration is translating CI configurations, which is challenging due to the intrinsic complexity of CI configurations and the need to understand semantic differences and relationships across CI platforms. With the advent of large language models (LLMs), recent advances in software engineering highlight their potential for CI configuration translation. In this paper, we present a study on LLM-based CI configuration translation, focusing on the migration from Travis CI to GitHub Actions. First, using 811 migration records, we quantify the effort involved and find that developers read an average of 38 lines of Travis configuration and write 58 lines of GitHub Actions configuration, with nearly half of the migrations requiring multiple commits. We further analyze translations produced by each of the four LLMs and identify 1,121 issues grouped into four categories: logic inconsistencies (38%), platform discrepancies (32%), environment errors (25%), and syntax errors (5%). Finally, we evaluate three enhancement strategies and show that combining guideline-based prompting with iterative refinement achieves the best performance, reaching a Build Success Rate of 75.5%-nearly a threefold improvement over GPT-4o with a basic prompt.

Problem

• Translating CI configurations between different platforms
• Addressing semantic differences in CI migration processes
• Improving accuracy of automated CI configuration translation

Innovation

• Using large language models for CI configuration translation
• Applying guideline-based prompting with iterative refinement
• Achieving 75.5% build success rate in migrations