🤖 AI Summary
This work investigates the implicit bias mechanism of mirror flow in wide-and-shallow neural networks for univariate least-squares regression. Using variational analysis and the theory of wide-network limits, we rigorously characterize the curvature-based regularization bias induced by mirror flow in function space. Our contributions are threefold: (1) For ReLU and absolute-value activated networks, mirror flow replicates the generalization bias of gradient flow—without skip connections or data preprocessing; (2) By introducing a scalable potential function, we transcend kernel-method limitations, enabling decoupled regularization of input location and curvature magnitude, and supporting rich, non-RKHS biases; (3) In the infinite-width limit, mirror flow and gradient flow exhibit identical implicit bias, revealing their shared “lazy training” regime. These results provide new theoretical insights and analytical tools for understanding implicit regularization in deep learning.
📝 Abstract
We examine the implicit bias of mirror flow in univariate least squares error regression with wide and shallow neural networks. For a broad class of potential functions, we show that mirror flow exhibits lazy training and has the same implicit bias as ordinary gradient flow when the network width tends to infinity. For ReLU networks, we characterize this bias through a variational problem in function space. Our analysis includes prior results for ordinary gradient flow as a special case and lifts limitations which required either an intractable adjustment of the training data or networks with skip connections. We further introduce scaled potentials and show that for these, mirror flow still exhibits lazy training but is not in the kernel regime. For networks with absolute value activations, we show that mirror flow with scaled potentials induces a rich class of biases, which generally cannot be captured by an RKHS norm. A takeaway is that whereas the parameter initialization determines how strongly the curvature of the learned function is penalized at different locations of the input space, the scaled potential determines how the different magnitudes of the curvature are penalized.