🤖 AI Summary
Concept drift detection is widely employed in data stream learning, yet its efficacy remains inadequately validated, and it often fails to distinguish genuine distributional shifts from spurious drifts induced by the detection mechanism itself. This work introduces the notion of the “window dilemma,” revealing that sliding-window–based detection is fundamentally ill-posed: observed drift may stem from windowing choices rather than actual changes in the underlying data-generating process. Through theoretical analysis, illustrative examples, and large-scale empirical comparisons, the study systematically evaluates a range of drift detectors against non-drift-aware adaptive and batch learning methods. The results demonstrate that conventional batch learners consistently outperform drift-detection–based streaming classifiers across most scenarios, thereby raising fundamental questions about the necessity and practical utility of prevailing concept drift detection paradigms.
📝 Abstract
Non-stationarity of an underlying data generating process that leads to distributional changes over time is a key characteristic of Data Streams. This phenomenon, commonly referred to as Concept Drift, has been intensively studied, and Concept Drift Detectors have been established as a class of methods for detecting such changes (drifts). For the most part, Drift Detectors compare regions (windows) of the data stream and detect drift if those windows are sufficiently dissimilar. In this work, we introduce the Window Dilemma, an observation that perceived drift is a product of windowing and not necessarily the underlying data generating process. Additionally, we highlight that drift detection is ill-posed, primarily because verification of drift events are implausible in practice. We demonstrate these contributions first by an illustrative example, followed by empirical comparisons of drift detectors against a variety of alternative adaptation strategies. Our main finding is that traditional batch learning techniques often perform better than their drift-aware counterparts further bringing into question the purpose of detectors in Stream Classification.