🤖 AI Summary
This paper addresses the challenge of quantifying uncertainty in cross-validation (CV) performance estimates—particularly the difficulty of distinguishing true performance differences from random fluctuations during model comparison. We propose an efficient and robust bootstrap-based method that decomposes the variance of CV estimates using a random-effects model, enabling valid statistical inference on CV performance differences without strong modeling assumptions. Compared to standard bootstrap, our approach substantially reduces computational cost while yielding confidence intervals with accurate coverage probability and high statistical power. Extensive evaluations—including simulations and real-world applications across classification, regression, and time-series tasks—demonstrate its strong robustness and generalizability under small-sample settings, non-independent CV folds, and heterogeneous data distributions. The method provides reliable uncertainty quantification to support principled model selection.
📝 Abstract
Cross-validation is a widely used technique for evaluating the performance of prediction models. It helps avoid the optimism bias in error estimates, which can be significant for models built using complex statistical learning algorithms. However, since the cross-validation estimate is a random value dependent on observed data, it is essential to accurately quantify the uncertainty associated with the estimate. This is especially important when comparing the performance of two models using cross-validation, as one must determine whether differences in error estimates are a result of chance fluctuations. Although various methods have been developed for making inferences on cross-validation estimates, they often have many limitations, such as stringent model assumptions This paper proposes a fast bootstrap method that quickly estimates the standard error of the cross-validation estimate and produces valid confidence intervals for a population parameter measuring average model performance. Our method overcomes the computational challenge inherent in bootstrapping the cross-validation estimate by estimating the variance component within a random effects model. It is just as flexible as the cross-validation procedure itself. To showcase the effectiveness of our approach, we employ comprehensive simulations and real data analysis across three diverse applications.
