🤖 AI Summary
This work elucidates the underlying mechanism of gated MLPs by offering the first explanation of their success through the lens of symmetry breaking. It demonstrates that a gated MLP can be interpreted as a rank-1 approximation of bilinear attention, where the query and key correspond to two distinct factors, and the nonlinear activation is applied exclusively to one factor. This asymmetric treatment breaks both the exchange symmetry between the two factors and the inverse scaling symmetry induced by non-homogeneous activation functions. The analysis establishes a theoretical connection between gated MLPs and attention mechanisms, clarifying the origin of their performance advantages and providing a principled foundation for designing novel, efficient architectures.
📝 Abstract
We show that the conventional gated MLP can be viewed as a rank-1 approximation to a bilinear attention mechanism with two distinct factors corresponding to the query and the key. We further show that moving the nonlinearity onto one factor breaks the exchange symmetry between the two factors and, for non-homogeneous activations, the inverse-scaling symmetry as well. This perspective may help explain why gated MLPs are effective in practice and inform the design of future architectures.