🤖 AI Summary
This paper investigates how parameter norm regularization affects functional sparsity—measured by the number of kinks—in single-hidden-layer ReLU networks. Method: We conduct theoretical analysis of minimum-norm interpolators, distinguishing cases where bias parameters are either unregularized or explicitly/implicitly regularized. Contribution/Results: When biases are unregularized, the representation cost of the minimum-norm interpolator is characterized by a weighted total variation (TV) with weight $sqrt{1+x^2}$, yielding non-unique and non-sparse solutions. In contrast, bias regularization eliminates this weighting, ensuring uniqueness and sparsity of the minimum-norm solution; its parameter norm equals exactly the TV of the target function’s second derivative. This work establishes, for the first time, an exact equivalence between the minimum parameter norm of univariate ReLU networks and the second-derivative TV of the learned function. Crucially, it identifies bias regularization as a necessary condition for inducing structural sparsity in such models.
📝 Abstract
Controlling the parameters' norm often yields good generalisation when training neural networks. Beyond simple intuitions, the relation between regularising parameters' norm and obtained estimators remains theoretically misunderstood. For one hidden ReLU layer networks with unidimensional data, this work shows the parameters' norm required to represent a function is given by the total variation of its second derivative, weighted by a $sqrt{1+x^2}$ factor. Notably, this weighting factor disappears when the norm of bias terms is not regularised. The presence of this additional weighting factor is of utmost significance as it is shown to enforce the uniqueness and sparsity (in the number of kinks) of the minimal norm interpolator. Conversely, omitting the bias' norm allows for non-sparse solutions. Penalising the bias terms in the regularisation, either explicitly or implicitly, thus leads to sparse estimators.
