🤖 AI Summary
This study addresses the prevalence and impact of non-deterministic (i.e., “flaky”) builds in GitHub Actions, which significantly undermine the reliability of continuous integration and waste computational resources. Leveraging re-execution data from 1,960 open-source Java projects, the work presents the first systematic characterization of flaky builds, revealing that 67.73% of re-run builds are affected, spanning 51.28% of the projects, and identifying 15 distinct root causes. To mitigate this issue, the authors propose a job-level machine learning approach for detecting flaky failures. Evaluated against state-of-the-art baselines, the method achieves up to a 20.3% improvement in F1-score, substantially enhancing the accuracy of flaky build identification.
📝 Abstract
Continuous Integration (CI) is widely used to provide rapid feedback on code changes; however, CI build outcomes are not always reliable. Builds may fail intermittently due to non-deterministic factors, leading to flaky builds that undermine developers'trust in CI, waste computational resources, and threaten the validity of CI-related empirical studies. In this paper, we present a large-scale empirical study of flaky builds in GitHub Actions based on rerun data from 1,960 open-source Java projects. Our results show that 3.2% of builds are rerun, and 67.73% of these rerun builds exhibit flaky behavior, affecting 1,055 (51.28%) of the projects. Through an in-depth failure analysis, we identify 15 distinct categories of flaky failures, among which flaky tests, network issues, and dependency resolution issues are the most prevalent. Building on these findings, we propose a machine learning-based approach for detecting flaky failures at the job level. Compared with a state-of-the-art baseline, our approach improves the F1-score by up to 20.3%.