What Happened in This Pipeline? Diffing Build Logs with CiDiff

2025-04-25

AI Summary

In CI regression debugging, verbose build logs severely hinder the efficiency of traditional text-difference algorithms (e.g., LCS) and impose heavy cognitive burdens on manual analysis. This paper proposes CiDiff—the first lightweight, semantic-aware differencing algorithm specifically designed for the structural characteristics of CI build logs. Its core contributions are: (1) syntax-aware log chunking to group semantically coherent lines; (2) build-phase-aware semantic alignment and weighted matching to prioritize critical failures; and (3) an optimized LCS variant that jointly preserves line-order stability and semantic consistency. Evaluated on 17,906 real-world CI regression cases, CiDiff reduces the median number of candidate diff lines by 60%. A user study with professional developers shows that 70% prefer CiDiff for debugging, while only 5% favor conventional LCS-based diff tools—demonstrating its practical efficacy and usability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Algorithm Configuration
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Continuous integration (CI) is widely used by developers to ensure the quality and reliability of their software projects. However, diagnosing a CI regression is a tedious process that involves the manual analysis of lengthy build logs. In this paper, we explore how textual differencing can support the debugging of CI regressions. As off-the-shelf diff algorithms produce suboptimal results, in this work we introduce a new diff algorithm specifically tailored to build logs called CiDiff. We evaluate CiDiff against several baselines on a novel dataset of 17 906 CI regressions, performing an accuracy study, a quantitative study and a user-study. Notably, our algorithm reduces the number of lines to inspect by about 60 % in the median case, with reasonable overhead compared to the state-of-practice LCS-diff. Finally, our algorithm is preferred by the majority of participants in 70 % of the regression cases, whereas LCS-diff is preferred in only 5 % of the cases.

Problem

• Diagnosing CI regressions via manual log analysis is tedious
• Existing diff algorithms perform poorly on build logs
• Proposing CiDiff to reduce inspected lines by 60%

Innovation

• Introduces CiDiff for build log differencing
• Reduces inspected lines by 60 percent
• Preferred by users in 70 percent cases