regularization techniques

This study addresses the challenge of selecting an optimal regularization method that balances predictive accuracy and feature selection stability based on data characteristics. Through systematic Monte Carlo simulations across a seven-dimensional parameter space—encompassing 134,400 experiments with eight production-grade models—the authors evaluate Ridge, Lasso, ElasticNet, and Post-Lasso OLS. They reveal, for the first time, that Lasso suffers severe recall degradation (as low as 0.18) under conditions of high multicollinearity and low signal-to-noise ratio (SNR), whereas ElasticNet remains robust (achieving a recall of 0.93). The work proposes practical selection guidelines based on sample size, feature correlation, and SNR, and demonstrates that when the sample-to-feature ratio is sufficiently large (n/p ≥ 78), mainstream methods exhibit comparable predictive performance.

Conventional fixed-weight penalty methods in deep learning struggle to simultaneously satisfy constraints and maintain model performance, while suffering from high hyperparameter tuning costs. Method: This paper proposes an end-to-end, constraint-first optimization paradigm. It systematically identifies the fundamental trade-off between constraint strictness and model performance inherent in standard penalty methods and introduces a differentiable augmented Lagrangian framework centered on adaptive Lagrange multipliers—enabling gradient backpropagation and automatic differentiation, and seamlessly integrating with PyTorch and TensorFlow. Contribution/Results: Evaluated across fairness, robustness, and causal constraint tasks, the method achieves 100% constraint satisfaction without compromising classification or regression accuracy, eliminates manual hyperparameter tuning, and improves training efficiency by 3.2×.

This paper investigates how parameter norm regularization affects functional sparsity—measured by the number of kinks—in single-hidden-layer ReLU networks. Method: We conduct theoretical analysis of minimum-norm interpolators, distinguishing cases where bias parameters are either unregularized or explicitly/implicitly regularized. Contribution/Results: When biases are unregularized, the representation cost of the minimum-norm interpolator is characterized by a weighted total variation (TV) with weight $sqrt{1+x^2}$, yielding non-unique and non-sparse solutions. In contrast, bias regularization eliminates this weighting, ensuring uniqueness and sparsity of the minimum-norm solution; its parameter norm equals exactly the TV of the target function’s second derivative. This work establishes, for the first time, an exact equivalence between the minimum parameter norm of univariate ReLU networks and the second-derivative TV of the learned function. Crucially, it identifies bias regularization as a necessary condition for inducing structural sparsity in such models.

This work addresses the fundamental trade-off between standard accuracy and adversarial robustness in supervised learning. Methodologically, it introduces the first architecture-level accuracy–robustness trade-off curve, quantifying the inverse relationship between these objectives across diverse neural network architectures; defines a sensitivity influence function to theoretically characterize the stability of optimal solutions under adversarial perturbations; and reveals—via theoretical analysis of overparameterized linear models—that adversarial training implicitly regularizes model dynamics, interpolating between L₁ (LASSO) and L₂ (ridge regression) behaviors. The approach integrates rigorous theoretical analysis, influence-function-based modeling, and extensive empirical evaluation across fully connected, deep, and varying-width networks. Results consistently validate the existence and structure of the trade-off, providing an interpretable, predictive theoretical foundation for principled neural architecture selection.

Existing theoretical frameworks struggle to characterize distribution-dependent generalization behavior of interpolation solutions in overparameterized models. Method: We introduce a distribution-dependent PAC-Chernoff bound—first enabling tight, precise generalization analysis of interpolation solutions—and define a computable model smoothness metric grounded in large-deviation theory. Building on this, we establish a unified theoretical framework linking regularization (ℓ₂, gradient penalty, initialization distance), data augmentation, and invariant architecture design to smoothness optimization. Results: We rigorously prove that prevalent training strategies—including weight decay, input gradient regularization, and data augmentation—implicitly enhance model smoothness. Our work provides the first distribution-dependent, tight, and interpretable theoretical foundation for the interpolation phenomenon in overparameterized learning, unifying empirical observations under a principled smoothness-centric lens.

This study addresses the limited generalization performance of standard norm-based regularization in neural networks when dealing with high-dimensional or feature-correlated settings, where conventional methods inadequately control model complexity. To overcome this, the authors propose two covariance-aware adaptive regularization techniques: first, incorporating the input feature covariance structure into ℓ₂ weight decay to refine ridge-type penalties; second, combining ℓ₁ sparsity with covariance-informed ℓ₂ regularization to achieve structured sparsity. By innovatively embedding feature covariance information into classical Lasso and ridge regression frameworks, the approach enables more precise complexity control. Extensive experiments on Monte Carlo simulations and real-world datasets—including building cooling load prediction and leukemia cell classification—demonstrate that the proposed methods significantly outperform traditional regularization strategies and substantially enhance model generalization.

This work addresses the challenge of achieving implicit complexity control in learning algorithms without relying on explicit regularization terms. It proposes a self-regularized learning framework that implicitly constrains predictors through the complexity of the simplest comparator, offering a unified characterization of the generalization behavior of algorithms such as gradient descent. The framework encompasses both classical regularization methods and implicit regularization mechanisms, and further enables data-driven hyperparameter selection. Theoretically, the authors establish a general self-regularization theory, derive minimax optimal convergence rates, and—by integrating early stopping within reproducing kernel Hilbert spaces (RKHS)—provide the first theoretical guarantee for data-dependent early stopping strategies.

The rule of thumb regarding the relationship between the bias-variance tradeoff and model size plays a key role in classical machine learning, but is now well-known to break down in the overparameterized setting as per the double descent curve. In particular, minimum-norm interpolating estimators can perform well, suggesting the need for new tradeoff in these settings. Accordingly, we propose a regularization-sharpness tradeoff for overparameterized linear regression with an $\ell^p$ penalty. Inspired by the interpolating information criterion, our framework decomposes the selection penalty into a regularization term (quantifying the alignment of the regularizer and the interpolator) and a geometric sharpness term on the interpolating manifold (quantifying the effect of local perturbations), yielding a tradeoff analogous to bias-variance. Building on prior analyses that established this information criterion for ridge regularizers, this work first provides a general expression of the interpolating information criterion for $\ell^p$ regularizers where $p \ge 2$. Subsequently, we extend this to the LASSO interpolator with $\ell^1$ regularizer, which induces stronger sparsity. Empirical results on real-world datasets with random Fourier features and polynomials validate our theory, demonstrating how the tradeoff terms can distinguish performant linear interpolators from weaker ones.

Existing methods often rely on soft penalties to approximate sample-level constraints, which struggle to strictly enforce hard requirements. This work proposes the first sample-wise constrained learning framework based on the sequential penalty method, enabling strict satisfaction of per-sample constraints within deep learning while providing convergence guarantees. By systematically integrating sequential penalty mechanisms into end-to-end training, the approach balances theoretical rigor with practical feasibility. Experiments on image processing tasks demonstrate that the proposed framework not only ensures strict adherence to constraints but also maintains competitive model performance.

This study investigates the generalization of over-parameterized neural networks that defy classical statistical learning theory by achieving strong performance despite having far more parameters than training samples. Through systematic PyTorch experiments on CIFAR-10 and MNIST with multiple random seeds, the work examines how optimization dynamics and implicit regularization—mediated by minibatch SGD behavior, loss landscape geometry, the neural tangent kernel (NTK) mechanism, double descent phenomena, and the lottery ticket hypothesis—affect generalization. The findings reveal that minibatch training converges to flatter minima, evidenced by an 11.8-fold reduction in the top Hessian eigenvalues, yielding a 1.61% improvement in test accuracy. Moreover, sparse subnetworks retaining only 10% of the original parameters incur less than 1.15% performance loss upon retraining. These results underscore the critical interplay among architecture, optimizer, and loss landscape geometry in enabling generalization in high-dimensional models.