🤖 AI Summary
This study addresses the challenge of model performance degradation caused by temporal shifts in feature distributions within real-world time-series data. The authors propose a parameter-free, automated detection method that leverages a regression model to predict sample timestamps and integrates feature importance analysis to identify time-sensitive features. By unifying the quantification of both univariate and multivariate distributional changes across numerical and categorical features, the approach offers strong scalability. Experimental results demonstrate that the method effectively and comprehensively captures a wide range of fundamental drift patterns on both real-world and synthetic datasets, achieving high detection accuracy while maintaining computational efficiency.
📝 Abstract
When working with real-world temporal data, it is common to encounter features whose distribution is changing over time. The naive employment of Machine Learning models on this unstable data might lead to rapidly degrading performance, especially if the new distribution is much different from what was previously seen during training. In order to cope with this problem, it is critical to automatically identify features that are changing over time. With these features detected, data scientists and other practitioners will be able to mitigate the issue (for instance, by applying data transformations), deploying more robust models that retain high performance for longer periods of time. In this paper, we describe which temporal changes a feature should not suffer from, and propose TEDD, a technique to a) identify when a dataset might lead to an unstable Machine Learning model and b) automatically detect which features cause such lack of robustness. In order to achieve it, we leverage a regression model to highlight which features contribute to a good prediction of an instance's timestamp. We compare our approach to other methods in real and synthetic data, testing their detection capability on all simple change patterns. We show that our method: detects all types of basic changes, both for numerical and categorical features; can detect multivariate drifts; returns a comparable value measuring the amount of change of each feature; requires no parameter tuning; and is scalable both on number of features and instances of the dataset.