🤖 AI Summary
This paper addresses the critical problem of concept drift detection in supervised learning. We propose a multivariate exponentially weighted moving average (MEWMA) control chart method based on shifts in the mean of Fisher score vectors. Our key contributions are: (1) integrating nested bootstrapping with the 0.632+ variance correction to accurately estimate control limits—thereby significantly improving false alarm rate (FAR) control, especially under small-sample and stringent FAR constraints; and (2) eliminating the need for an initial labeled dataset to calibrate control limits, enabling full utilization of all available samples for model training and thus jointly enhancing detection sensitivity and model accuracy. Empirical evaluation demonstrates that our method achieves superior FAR control precision compared to conventional approaches, with particularly pronounced advantages in small-sample regimes and under strict false alarm constraints.
📝 Abstract
Monitoring for changes in a predictive relationship represented by a fitted supervised learning model (aka concept drift detection) is a widespread problem, e.g., for retrospective analysis to determine whether the predictive relationship was stable over the training data, for prospective analysis to determine when it is time to update the predictive model, for quality control of processes whose behavior can be characterized by a predictive relationship, etc. A general and powerful Fisher score-based concept drift approach has recently been proposed, in which concept drift detection reduces to detecting changes in the mean of the model's score vector using a multivariate exponentially weighted moving average (MEWMA). To implement the approach, the initial data must be split into two subsets. The first subset serves as the training sample to which the model is fit, and the second subset serves as an out-of-sample test set from which the MEWMA control limit (CL) is determined. In this paper, we develop a novel bootstrap procedure for computing the CL. Our bootstrap CL provides much more accurate control of false-alarm rate, especially when the sample size and/or false-alarm rate is small. It also allows the entire initial sample to be used for training, resulting in a more accurate fitted supervised learning model. We show that a standard nested bootstrap (inner loop accounting for future data variability and outer loop accounting for training sample variability) substantially underestimates variability and develop a 632-like correction that appropriately accounts for this. We demonstrate the advantages with numerical examples.