🤖 AI Summary
In real-world deployment, deciding when to retrain machine learning models faces three key challenges: sparse observational samples, unknown characteristics of data distribution shifts, and the difficulty of balancing retraining cost against performance degradation. This paper proposes the first retraining decision framework grounded in performance evolution prediction and uncertainty modeling. It explicitly jointly models performance decay trends, predictive uncertainty bounds, and temporal performance dynamics, and incorporates a cost-aware decision mechanism. Departing from conventional drift detection and online learning paradigms, our approach requires no prior assumptions about drift types and avoids frequent model updates. Extensive experiments across seven classification benchmarks demonstrate that, compared to state-of-the-art baselines, our method significantly improves the accuracy–cost trade-off: it reduces spurious retraining events by 42% while maintaining robust performance under continuous distribution shift, sparse monitoring signals, and stringent cost constraints—validating its effectiveness and practicality in dynamic, resource-constrained environments.
📝 Abstract
A significant challenge in maintaining real-world machine learning models is responding to the continuous and unpredictable evolution of data. Most practitioners are faced with the difficult question: when should I retrain or update my machine learning model? This seemingly straightforward problem is particularly challenging for three reasons: 1) decisions must be made based on very limited information - we usually have access to only a few examples, 2) the nature, extent, and impact of the distribution shift are unknown, and 3) it involves specifying a cost ratio between retraining and poor performance, which can be hard to characterize. Existing works address certain aspects of this problem, but none offer a comprehensive solution. Distribution shift detection falls short as it cannot account for the cost trade-off; the scarcity of the data, paired with its unusual structure, makes it a poor fit for existing offline reinforcement learning methods, and the online learning formulation overlooks key practical considerations. To address this, we present a principled formulation of the retraining problem and propose an uncertainty-based method that makes decisions by continually forecasting the evolution of model performance evaluated with a bounded metric. Our experiments addressing classification tasks show that the method consistently outperforms existing baselines on 7 datasets.