On Bootstrapping Lasso in Generalized Linear Models

2024-03-28

AI Summary

For generalized linear models (GLMs), the Lasso estimator lacks a closed-form asymptotic distribution, rendering conventional asymptotic inference invalid. To address this, we propose a unified and theoretically rigorous bootstrap framework based on residual resampling followed by refitting with Lasso-penalized log-likelihood optimization. We establish finite-sample consistency of the proposed bootstrap procedure under mild regularity conditions, overcoming the nondifferentiability barrier identified by Knight & Fu (2000). Simulation studies demonstrate that, even at moderate sample sizes, our method yields confidence intervals with coverage probabilities closer to nominal levels and substantially improved statistical power for hypothesis testing. Empirical analysis on real data confirms its robustness and practical utility. To our knowledge, this is the first inference framework for the Lasso in GLMs that is simultaneously general—applicable across all GLM subfamilies—computationally feasible, and supported by rigorous theoretical guarantees.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Inference
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Generalized linear models or GLM constitute plethora of sub-models which extends the ordinary linear regression by connecting the mean of response variable with the covariates through appropriate link functions. On the other hand, Lasso is a popular and easy-to-implement penalization method in regression when not all covariates are relevant. However, Lasso does not generally have a tractable asymptotic distribution (Knight and Fu (2000)). In this paper, we develop a Bootstrap method which works as an alternative to the asymptotic distribution of Lasso for all the submodels of GLM. We support our theoretical findings by showing good finite-sample properties of the proposed Bootstrap method through a moderately large simulation study. We also implement our method on a real data set.

Problem

• Develop Bootstrap method for Lasso in GLM
• Provide alternative to Lasso's asymptotic distribution
• Validate method with simulations and real data

Innovation

• Bootstrap method for Lasso in GLM
• Alternative to asymptotic distribution
• Good finite-sample properties demonstrated