Using Large Language Models to Support Automation of Failure Management in CI/CD Pipelines: A Case Study in SAP HANA

2026-02-06

AI Summary

This work addresses the long-standing reliance on manual intervention in CI/CD pipeline failure management, which suffers from low efficiency and limited automation capabilities, particularly when handling unstructured failure information. To overcome these challenges, the authors propose an automated approach that integrates large language models (LLMs) with domain-specific knowledge—including historical failure data, pipeline context, and repair instructions—enabling end-to-end failure localization and repair generation for the first time in the large-scale industrial project SAP HANA. Experimental results demonstrate that, when augmented with domain knowledge, the method achieves a 97.4% accuracy in failure localization and generates fully correct repair solutions in 92.1% of cases, substantially outperforming knowledge-agnostic baselines. Ablation studies further confirm the critical contribution of historical failure data to overall performance gains.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

CI/CD pipeline failure management is time-consuming when performed manually. Automating this process is non-trivial because the information required for effective failure management is unstructured and cannot be automatically processed by traditional programs. With their ability to process unstructured data, large language models (LLMs) have shown promising results for automated failure management by previous work. Following these studies, we evaluated whether an LLM-based system could automate failure management in a CI/CD pipeline in the context of a large industrial software project, namely SAP HANA. We evaluated the ability of the LLM-based system to identify the error location and to propose exact solutions that contain no unnecessary actions. To support the LLM in generating exact solutions, we provided it with different types of domain knowledge, including pipeline information, failure management instructions, and data from historical failures. We conducted an ablation study to determine which type of domain knowledge contributed most to solution accuracy. The results show that data from historical failures contributed the most to the system's accuracy, enabling it to produce exact solutions in 92.1% of cases in our dataset. The system correctly identified the error location with 97.4% accuracy when provided with domain knowledge, compared to 84.2% accuracy without it. In conclusion, our findings indicate that LLMs, when provided with data from historical failures, represent a promising approach for automating CI/CD pipeline failure management.

Problem

• CI/CD pipeline
• failure management
• automation
• unstructured data
• large language models

Innovation

• Large Language Models
• CI/CD Pipeline
• Failure Management
• Historical Failure Data
• Automated Debugging