🤖 AI Summary
This study addresses the challenge of alarm fatigue in continuous model monitoring caused by high false positive rates of existing drift detectors, which undermines monitoring reliability. It presents the first systematic evaluation of the cumulative false positive behavior of five widely used methods—Population Stability Index (PSI), Kolmogorov–Smirnov (KS) test, Maximum Mean Discrepancy (MMD), Least-Squares Density Difference (LSDD), and adversarial validation—under continuous monitoring settings, incorporating Bonferroni correction for multiple hypothesis testing. The empirical analysis reveals that PSI exhibits markedly improved stability when sample sizes exceed 200, whereas KS, MMD, and LSDD demonstrate greater reliability with smaller batch sizes. While Bonferroni correction effectively suppresses false positives, it concurrently reduces detection sensitivity. These findings offer practical guidance for selecting batch sizes and calibrating detectors in real-world deployments, balancing robustness and responsiveness.
📝 Abstract
Drift detection is a core component of production machine learning monitoring systems, where detectors are used to compare incoming data with a reference distribution and trigger alerts when changes occur. However, these detectors are often evaluated in research settings that emphasize detection accuracy under synthetic shifts, while overlooking false alarms under continuous monitoring. In production environments, models are monitored repeatedly over time and across many features, and even small false positive rates can accumulate into frequent alerts, leading to alarm fatigue. We empirically analyze false positive behavior across five commonly used drift detectors: PSI, KS, MMD, LSDD, and adversarial validation. Consistent with existing literature, PSI exhibits strong sensitivity to batch size, producing frequent false alarms at small sample sizes; however, we further observe that its behavior stabilizes and improves substantially once batch sizes exceed approximately 200 samples. In contrast, KS, MMD, and LSDD display persistent fluctuations across batch sizes, while remaining comparatively more reliable than PSI in low-data regimes. Applying a Bonferroni correction reduces false positive rates, but often at the cost of reduced true positive sensitivity, reinforcing the well-known stability - sensitivity trade-off in drift detection. This work provides a systematic comparison of false positive behavior across multiple drift detectors under continuous monitoring conditions. We identify tradeoffs across detector families and provide practical guidelines for selecting and calibrating drift detectors in production ML systems.