Choosing the Right Regularizer for Applied ML: Simulation Benchmarks of Popular Scikit-learn Regularization Frameworks

2026-04-03

AI Summary

This study addresses the challenge of selecting an optimal regularization method that balances predictive accuracy and feature selection stability based on data characteristics. Through systematic Monte Carlo simulations across a seven-dimensional parameter space—encompassing 134,400 experiments with eight production-grade models—the authors evaluate Ridge, Lasso, ElasticNet, and Post-Lasso OLS. They reveal, for the first time, that Lasso suffers severe recall degradation (as low as 0.18) under conditions of high multicollinearity and low signal-to-noise ratio (SNR), whereas ElasticNet remains robust (achieving a recall of 0.93). The work proposes practical selection guidelines based on sample size, feature correlation, and SNR, and demonstrates that when the sample-to-feature ratio is sufficiently large (n/p ≥ 78), mainstream methods exhibit comparable predictive performance.

Technology Category

• Machine Learning: Dimensionality Reduction/Feature Selection
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

This study surveys the historical development of regularization, tracing its evolution from stepwise regression in the 1960s to recent advancements in formal error control, structured penalties for non-independent features, Bayesian methods, and l0-based regularization (among other techniques). We empirically evaluate the performance of four canonical frameworks -- Ridge, Lasso, ElasticNet, and Post-Lasso OLS -- across 134,400 simulations spanning a 7-dimensional manifold grounded in eight production-grade machine learning models. Our findings demonstrate that for prediction accuracy when the sample-to-feature ratio is sufficient (n/p >= 78), Ridge, Lasso, and ElasticNet are nearly interchangeable. However, we find that Lasso recall is highly fragile under multicollinearity; at high condition numbers (kappa) and low SNR, Lasso recall collapses to 0.18 while ElasticNet maintains 0.93. Consequently, we advise practitioners against using Lasso or Post-Lasso OLS at high kappa with small sample sizes. The analysis concludes with an objective-driven decision guide to assist machine learning engineers in selecting the optimal scikit-learn-supported framework based on observable feature space attributes.

Problem

• regularization
• Lasso
• ElasticNet
• multicollinearity
• prediction accuracy

Innovation

• regularization
• multicollinearity
• ElasticNet
• simulation benchmark
• feature selection