🤖 AI Summary
This work proposes an iterative algorithm that automatically computes a near-optimal regularization strength for ridge regression under fixed design matrices, assuming limited samples, bounded covariance, and isotropic noise. The method leverages parameters from a generative model to construct, for the first time in the fixed-X setting, a provably convergent numerical procedure that achieves strong generalization to random designs through sample-based estimation. Experimental results demonstrate that, in both under- and over-parameterized regimes, the approach attains near-optimal generalization performance across varying sample sizes, dimensionality ratios, and noise levels, requiring only one or two additional ridge regression computations beyond the initial fit.
📝 Abstract
We consider $L^2$-regularized linear (ridge) regression over a finite data sample $X$ with bounded covariance and linear prediction targets $y$ with additive isotropic noise of finite variance. We present an iterative procedure to compute the optimal regularization strength numerically from the generative parameters in the fixed-$X$ setting and prove its convergence at limited noise levels. Our experimental evaluation over synthetic data shows that the proposed procedure combined with sample-based parameter estimates attains near-optimal random-$X$ generalization across a wide range of sample sizes, aspect ratios, and noise levels, at an added computational cost equivalent to one preliminary ridge regression in the underparameterized regime and two in the overparameterized case.