Doc2CI: A Multi-Service Study of CI Configuration Generation Using Large Language Models

2026-08-02

AI Summary

Automatically generating YAML configuration files that are both structurally valid and compliant with multiple continuous integration (CI) service specifications remains a significant challenge, and the capabilities of current large language models (LLMs) on this task are not well understood. This work introduces DOC2CI, the first cross-CI benchmark dataset comprising 3,363 document–YAML pairs, and systematically evaluates 14 open-source models alongside GPT-series models. A novel failure taxonomy is proposed to uncover the root causes of model discrepancies, and this study provides the first empirical evidence that document similarity and structural validity constitute distinct optimization objectives. Experiments reveal that even the largest models achieve an Exact Match rate below 3.1%; while 97% of generated outputs are syntactically parseable, only 71% conform to the target service schema. Schema-guided post-hoc repair without additional training boosts structural validity to 94%, whereas fine-tuning improves document similarity at the expense of standalone structural correctness.

Abstract

Adopting Continuous Integration (CI) often requires writing YAML configurations that are error-prone and challenging to maintain. Despite increasing LLM use in software engineering, their ability to generate CI configurations from natural language across services and model families remains unclear. This paper presents a large empirical study on using LLMs to generate CI configurations. We introduce DOC2CI, a benchmark of 3,363 description-to-YAML pairs collected from the official documentation of four CI services, and evaluate 14 open-weight models from 7B-34B parameters together with GPT-4o and GPT-4.1, producing over 53,000 configurations. We assess both reference alignment and schema validity to determine whether the generated configurations are structurally valid. We further develop a failure taxonomy from a manual analysis of 385 configurations and examine why LLMs disagree. Across models and services, exact reference reproduction never exceeds 3.1%, and while 97% of outputs parse as YAML, only 71% satisfy service schemas. Larger models improve structural validity, but code specialization provides no consistent advantage over comparable general models. Model differences are driven largely by output completeness: for the same request, some models generate the expected fragment while others produce a full workflow. Finally, a training-free schema-guided repair method improves schema validity to 94%, while fine-tuning improves similarity to documentation but reduces standalone validity. This suggests that similarity and validity are distinct objectives for CI generation and motivate schema-aware evaluation and tooling for LLM-based configuration generation.

Problem

• Continuous Integration
• LLM
• YAML configuration
• schema validity
• configuration generation

Innovation

• CI configuration generation
• large language models
• schema validity
• benchmark dataset
• schema-guided repair