Effects of width-dependent model hyperparameters and $\ell_2$-regularization on the loss landscape of two-layer ReLU networks

2026-07-18

AI Summary

This work investigates the loss landscape and global minima behavior of two-layer ReLU networks with ℓ² regularization (weight decay) across varying network widths. Through theoretical analysis and empirical validation, the authors derive sufficient conditions under which global minima collapse to the zero solution and obtain closed-form expressions for globally optimal parameters in the single-input-dimension setting. The study innovatively demonstrates that AdamW effectively prevents such parameter collapse, whereas SGD does not. Furthermore, it establishes that ℓ² regularization exhibits a width-independent connectivity-preserving effect and an increasingly pronounced dimensionality-reduction property as network width grows. These findings provide theoretical insights into the interplay between regularization and optimizers in over-parameterized neural networks.

Abstract

Understanding deep neural networks remains a central challenge in machine learning. In particular, the theoretical properties of even two-layer ReLU networks, especially in the presence of weight decay, remain poorly understood. To this end, we derive a sufficient condition on the hyperparameter settings under which the global minima collapse to the zero solution. Interestingly, our experiments reveal that using AdamW as an optimizer prevents the collapse of the learned parameters, whereas using SGD does not, which may help explain the success of AdamW in deep learning training. In addition, when restricting the input dimension to one, we derive an analytical solution for the globally optimal parameter sets of two-layer ReLU networks and show that $\ell_2$-regularization has a width-invariant effect on connectivity, but its dimensionality-reducing effect becomes stronger as the network width increases. These results provide insight into how width-dependent hyperparameters influence the geometry of regularized loss landscapes.

Problem

• ReLU networks
• loss landscape
• ℓ2-regularization
• network width
• global minima

Innovation

• ReLU networks
• ℓ²-regularization
• loss landscape
• width-dependent hyperparameters
• AdamW