🤖 AI Summary
This work addresses the inefficiency in traditional neural network optimization, where additive weight updates induce imbalanced relative perturbations across weights of differing magnitudes. To mitigate this, the authors propose a hybrid exponential-linear reparameterization of weights that integrates a sign-aware symmetric exponential pathway with an identity linear pathway. This construction, augmented with learnable scale, curvature, and offset parameters, induces a curved weight geometry wherein optimization step sizes scale proportionally with weight magnitudes. Coupled with a mismatched initialization strategy to encourage early symmetry breaking, the method achieves equivalent validation loss on OpenWebText using 1.32–1.49× fewer training steps across various Transformer architectures, with particularly pronounced gains for wide models.
📝 Abstract
Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps. Adaptive optimizers such as Adam normalize updates per coordinate, but update steps remain additive; weights with very different magnitudes receive similarly sized absolute changes, producing very different relative perturbations. We introduce \textbf{\method} (\textbf{\methodshort}), a weight reparameterization for neural networks that combines a sign-aware symmetric-exponential pathway with an identity-like linear pathway. The symmetric-exponential pathway is near-linear for small raw weights but increasingly curved at larger magnitudes. Additive updates in logarithmic space map to magnitude-proportional changes in effective weight space. The linear pathway provides a direct route through the transform that we hypothesize stabilizes optimization, while learnable scale, curvature, and offset parameters control balance between pathways and the curvature of the exponential pathway. These components create a curved parameter-space geometry that empirically improves speed of loss descent over standard linear parameterization. We also identify a useful \emph{mismatched initialization}: raw weights are chosen so a symmetric version of the transform matches Xavier statistics, but training uses an asymmetric forward transform that leaves positive weights at full strength while making negative weights smaller in magnitude; in small-model ablations, this improves early optimization and may act as a form of symmetry breaking. We train transformers on OpenWebText over nine width$\times$depth configurations, \methodshort reaches matched validation loss in 1.32--1.49$\times$ fewer training steps, with the largest widths seeing the biggest gains.