Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasets

2026-09-20

AI Summary

研究针对大规模数据集上L1正则化逻辑模型的训练问题，发现较旧的方法比所谓的最新方法更有效，并推荐了一种基于LBFGS的新简单基线方法。

Abstract

Linear models with an $L_1$-norm penalty remain state-of-the-art for high-dimensional ($d > 1,000,000$) tasks, offering a straightforward method for solving real-world industry problems. Despite their widespread use in industry and utility, many $L_1$ solvers are not effective for general use, are prohibitively slow, and are ineffective in parallelization. This makes them difficult to train in an MLOps pipeline on large industry-scale corpora. In this work, we test several proposed ``state-of-the-art'' solutions from the literature and find that older methods are currently far superior for general use. We also identify several recommendations for academics to perform research that avoids erroneously overconfident results, which can prevent the transition to production use. Equally surprising, we find that a new and simple baseline, using LBFGS on a sub-gradient, is highly effective with minor tweaks, despite being dismissed in the literature for theoretical non-convergence. In practice, we find it is an easier-to-support and easier-to-scale method for production use.

Problem

• L1 Penalized Logistic Models
• Large Datasets
• MLOps pipeline
• High-dimensional tasks
• Parallelization

Innovation

• L1 penalized logistic models
• large datasets
• LBFGS on sub-gradient
• production use