From Cross-Validation to SURE: Asymptotic Risk of Tuned Regularized Estimators

2026-03-20

AI Summary

This study investigates the asymptotic out-of-sample prediction risk of regularized empirical risk minimization estimators tuned via cross-validation. By establishing an asymptotic equivalence between $n$-fold cross-validation and Stein’s Unbiased Risk Estimate (SURE), the work demonstrates for the first time that, in high-dimensional regularized regression, the tuning parameter selected by cross-validation converges almost surely to the global minimizer of SURE, which exhibits universal separability. Consequently, the prediction risk of the cross-validated estimator is asymptotically equivalent to that of the SURE-tuned shrinkage estimator. This equivalence yields a refined, data-adaptive characterization of prediction risk under the normal means model, substantially sharpening traditional worst-case risk bounds.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

We derive the asymptotic risk function of regularized empirical risk minimization (ERM) estimators tuned by $n$-fold cross-validation (CV). The out-of-sample prediction loss of such estimators converges in distribution to the squared-error loss (risk function) of shrinkage estimators in the normal means model, tuned by Stein's unbiased risk estimate (SURE). This risk function provides a more fine-grained picture of predictive performance than uniform bounds on worst-case regret, which are common in learning theory: it quantifies how risk varies with the true parameter. As key intermediate steps, we show that (i) $n$-fold CV converges uniformly to SURE, and (ii) while SURE typically has multiple local minima, its global minimum is generically well separated. Well-separation ensures that uniform convergence of CV to SURE translates into convergence of the tuning parameter chosen by CV to that chosen by SURE.

Problem

• asymptotic risk
• regularized ERM
• cross-validation
• SURE
• predictive performance

Innovation

• cross-validation
• SURE
• asymptotic risk
• regularized ERM
• uniform convergence