Neural Networks Provably Learn Spectral Representations for Group Composition

2026-06-01

AI Summary

This work investigates how neural networks spontaneously develop structured internal representations when trained to predict the composition of group elements. By lifting the training dynamics into the Fourier domain and integrating tools from group representation theory, Riemannian gradient flows, and dynamical systems analysis, the authors rigorously establish—for the first time—that neurons in a two-layer network, under random initialization, almost surely converge to a single irreducible representation. Moreover, cross-layer Fourier coefficients exhibit a rotationally aligned rank-one structure, revealing a low-rank compression mechanism inherent to group representations. In the case of Abelian groups, the theory fully characterizes the emergent population behavior: nontrivial representations diversify uniformly and generate Haar-distributed phases, which exponentially fast align through phase synchronization and representation competition, ultimately approximating indicator functions via majority voting.

Abstract

Understanding how structured internal structure emerges during neural network training is central to the study of deep learning. We investigate this phenomenon through the group composition task, where a two-layer neural network is trained to predict $g_1 \star g_2$ for elements of a finite group $G$. By lifting the projected gradient flow to the Fourier domain, we demonstrate that the training dynamics are governed by a Riemannian gradient ascent on a representation-theoretic energy functional. We prove that, under random initialization, this flow drives each neuron to converge almost surely toward a single irreducible representation, while the cross-layer Fourier coefficients achieve a rotational rank-one alignment. This framework provides a representation-theoretic account of feature learning and characterizes a novel low-rank compression phenomenon for matrix-valued group representations. Moreover, for Abelian groups, we provide a complete population-level description: random initialization promotes uniform diversification across nontrivial representations and induces Haar-uniform phases, jointly approximating the indicator via a majority-vote mechanism. We further prove that both phase alignment and representation competition emerge with exponential convergence rates.

Problem

• neural networks
• spectral representations
• group composition
• representation theory
• feature learning

Innovation

• representation theory
• Fourier domain
• neural network dynamics
• low-rank compression
• group composition