Lasso-Ridge Refitting: A Two-Stage Estimator for High-Dimensional Linear Regression

2025-12-11

AI Summary

To address the substantial estimation bias and high prediction error of the Lasso in high-dimensional linear regression, this paper proposes a two-stage Lasso-Ridge refitting method: the first stage employs Lasso for variable selection, and the second stage applies Ridge regularization on the selected submodel to correct estimation bias. This constitutes the first systematic framework that simultaneously achieves variable selection consistency and prediction consistency. Theoretical analysis establishes an upper bound on the prediction error and demonstrates that the proposed method strictly dominates the standard Lasso—even under its optimal tuning rate. Monte Carlo simulations show that the method reduces prediction error by 15–30% across diverse high-dimensional settings, significantly improves estimation accuracy, and maintains robust variable selection and generalization performance under challenging scenarios such as low signal-to-noise ratios and highly correlated designs.

Technology Category

Machine Learning: Ensemble MethodsSearch and Optimization: Learning to SearchIntelligent Robots: State Estimation

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphsSocial Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

The least absolute shrinkage and selection operator (Lasso) is a popular method for high-dimensional statistics. However, it is known that the Lasso often has estimation bias and prediction error. To address such disadvantages, many alternatives and refitting strategies have been proposed and studied. This work introduces a novel Lasso--Ridge method. Our analysis indicates that the proposed estimator achieves improved prediction performance in a range of settings, including cases where the Lasso is tuned at its theoretical optimal rate (sqrt{log(p)/n}). Moreover, the proposed method retains several key advantages of the Lasso, such as prediction consistency and reliable variable selection under mild conditions. Through extensive simulations, we further demonstrate that our estimator outperforms the Lasso in both prediction and estimation accuracy, highlighting its potential as a powerful tool for high-dimensional linear regression.

Problem

• Reduces Lasso's estimation bias and prediction error
• Improves prediction performance in high-dimensional linear regression
• Retains Lasso's advantages like variable selection consistency

Innovation

• Two-stage Lasso-Ridge estimator reduces bias
• Retains Lasso's variable selection and consistency
• Outperforms Lasso in prediction and estimation accuracy