Does $K$-fold CV based penalty perform variable selection or does it lead to $n^{1/2}$-consistency in Lasso?

2025-07-16

AI Summary

This paper investigates the statistical properties of Lasso tuning via fixed-K-fold cross-validation (CV) in fixed-dimensional heteroscedastic linear regression. We establish that the resulting Lasso estimator is $n^{1/2}$-consistent but fails to achieve variable selection consistency—a rigorous, first-time characterization of the fundamental trade-off between estimation consistency and model selection consistency under K-fold CV. Furthermore, we prove that bootstrap-based inference constructed from this estimator is asymptotically valid, providing theoretical justification for standard error estimation and confidence interval construction. Simulation studies corroborate the theoretical findings, demonstrating that while K-fold CV sacrifices variable selection accuracy, it robustly supports valid parametric inference. Our work fills a critical gap in the theoretical understanding of inference validity for CV-tuned Lasso estimators and offers practical guidance for high-dimensional statistical inference.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Inference
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Least absolute shrinkage and selection operator or Lasso, introduced by Tibshirani (1996), is one of the widely used regularization methods in regression. It is observed that the properties of Lasso vary wildly depending on the choice of the penalty parameter. The recent results of Lahiri (2021) suggest that, depending on the nature of the penalty parameter, Lasso can either be variable selection consistent or be $n^{1/2}-$consistent. However, practitioners generally implement Lasso by choosing the penalty parameter in a data-dependent way, the most popular being the $K$-fold cross-validation. In this paper, we explore the variable selection consistency and $n^{1/2}-$consistency of Lasso when the penalty is chosen based on $K$-fold cross-validation with $K$ being fixed. We consider the fixed-dimensional heteroscedastic linear regression model and show that Lasso with $K$-fold cross-validation based penalty is $n^{1/2}-$consistent, but not variable selection consistent. We also establish the $n^{1/2}-$consistency of the $K$-fold cross-validation based penalty as an intermediate result. Additionally, as a consequence of $n^{1/2}-$consistency, we establish the validity of Bootstrap to approximate the distribution of the Lasso estimator based on $K-$fold cross-validation. We validate the Bootstrap approximation in finite samples based on a moderate simulation study. Thus, our results essentially justify the use of $K$-fold cross-validation in practice to draw inferences based on $n^{1/2}-$scaled pivotal quantities in Lasso regression.

Problem

• Investigates if K-fold CV-based Lasso ensures variable selection consistency.
• Examines whether K-fold CV penalty achieves n^(1/2)-consistency in Lasso.
• Validates Bootstrap for inference with K-fold CV-based Lasso estimators.

Innovation

• Uses K-fold CV for Lasso penalty selection
• Shows n^(1/2)-consistency without variable selection
• Validates Bootstrap for Lasso estimator distribution