Upper Confidence Bounds for the Prediction Error of Kernel Ridge Regression via Gaussian Refitting

2026-07-30

AI Summary

This work addresses the challenge of constructing tight, computable upper confidence bounds for prediction errors in kernel ridge regression when noise variance is large or moment conditions are absent—scenarios where cross-validation suffers from the $n^{-1/2}$ convergence bottleneck. The authors propose a novel Gaussian refitting-based approach that, for the first time, integrates Anderson’s inequality with order statistics to calibrate confidence bounds at arbitrary levels without requiring moment assumptions. The method ensures that the fitted perturbation is monotonic in the noise magnitude and achieves the minimax-optimal contraction rate of $O_P(n^{-2s/(2s+1)})$. Empirical results demonstrate that its 95% confidence bounds contain the true error within a factor of two, whereas cross-validation can exhibit biases hundreds of times larger under heavy-tailed noise.

Abstract

Assessing a single model fit requires a computable upper confidence bound for the gap between the fit and the unknown truth, as mean estimates ignore realization variance. Standard cross-validation margins are bottlenecked at order $n^{-1/2}$ by noise fluctuations, even when the true error shrinks faster. While wild refitting cancels this noise level, existing Rademacher sign methods degenerate for kernel ridge regression and rely on unobservable quantities. We propose a Gaussian refit for kernel ridge regression. By Anderson's inequality, the fit movement is monotone in the noise sizes, yielding a computable tail bound. Assuming only symmetric noise, the bound requires no moment assumptions and is calibrated at any confidence level via order statistics. Theoretically, using a worst-case envelope, the bound contracts at the minimax rate $O_P(n^{-2s/(2s+1)})$, correctly matching the prediction error. Empirically, using a practical data-driven envelope, the bound maintains full coverage within twice the true $95\%$ error quantile. By contrast, cross-validation exceeds this quantile by factors up to $51$, and by hundreds under infinite-variance noise. The procedure extends empirically to nonlinear constrained estimators and real spatial data.

Problem

• Kernel Ridge Regression
• Prediction Error
• Upper Confidence Bound
• Gaussian Refitting
• Model Assessment

Innovation

• Gaussian refitting
• upper confidence bound
• kernel ridge regression
• prediction error
• minimax rate