Evaluating Change Point Detection Methods for Software Performance Regression Analysis

2026-10-06

AI Summary

This study addresses the lack of systematic guidance for selecting change point detection (CPD) methods in software performance regression monitoring. Leveraging real-world time-series data and manually annotated benchmarks, it presents the first comprehensive quantitative evaluation of twelve CPD algorithms for identifying abrupt software performance changes. Through large-scale empirical analysis and annotation consistency validation, the research reveals significant differences among these methods in detection accuracy and scenario applicability. This work fills a critical evaluation gap regarding the application of CPD techniques within software engineering, providing essential empirical evidence and practical guidance for algorithm selection in automated performance regression monitoring systems.

Abstract

Performance issues in software systems are a critical quality issue that can erode user trust, violate service-level agreements, and ultimately affect business efficiency. Consequently, software performance engineering has shifted its focus to developing robust techniques to detect performance regressions as early as possible in the development cycle. Performance regression analysis often relies on time series of performance measurements to detect significant changes in performance behavior. Change Point Detection (CPD) methods have been widely used to automate the identification of such changes in various domains, including finance, healthcare, and performance monitoring. However, the effectiveness of these methods for software performance measurements has not been thoroughly evaluated. In this paper, we present a comprehensive study to evaluate the effectiveness of various CPD methods on real-world software performance measurement datasets. We start by collecting performance measurement data from three large software systems and characterizing the unique properties of performance time series data. Then, we undertake a large-scale effort to annotate and evaluate the consistency of human annotators' identification of potential performance changes. Thereafter, we evaluate the accuracy of twelve distinct CPD methods in detecting potential performance changes, providing insights into their applicability and effectiveness in software performance regression analysis.

Problem

• Change Point Detection
• Software Performance Regression
• Time Series Analysis
• Performance Monitoring

Innovation

• Change Point Detection
• Performance Regression Analysis
• Software Performance Engineering
• Time Series Data
• Method Evaluation