Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky

2026-09-30

AI Summary

This study addresses why language model concept embeddings exhibit specific geometric structures, such as circular or saddle-shaped configurations. Drawing upon group theory and harmonic analysis, this work reveals the intrinsic mechanism by which the statistical symmetries of data determine embedding geometry. By deriving the correspondence between irreducible representations and Fourier modes, the proposed approach generalizes translational symmetry to arbitrary finite groups and homogeneous spaces, establishing a unified theoretical framework that explains embedding geometries across diverse domains. The framework successfully reproduces known embedding structures, including the circular arrangement of months, the circle of fifths for musical chords, and spherical harmonics in astronomical contexts. These results validate the universality of the proposed theory across multimodal data spanning music, astronomy, and other fields.

Abstract

The representations that language models learn for concepts such as months, weekdays, and places display consistent geometric structure: circles and saddle-shaped "Pringle" manifolds. Recent work traced these structures to $\textit{translation symmetry}$ in word co-occurrence statistics, deriving the observed Fourier geometry when co-occurrence depends only on distance on an abelian lattice of concepts. We demonstrate that more general notions of symmetry lead to equally structured predictions. Considering symmetries defined by arbitrary finite groups, compact groups, and homogeneous spaces, we prove that whenever the co-occurrence statistics of a word family are invariant under a group $G$, the learned word embeddings consist of matrix elements of the irreducible representations (irreps) of $G$. Circles and Pringles arise when $G$ is cyclic, in which case the irreps are Fourier modes. We verify the irrep structure in three experimental settings. (i) The cyclic group $\mathbb{Z}_{12}$: for the months of the year we recover the known circular geometry. (ii) A dihedral group acting on the major and minor triads: we unify two classical observations -- that transposition and chord inversion form a group ($T/I$) acting on chords (music theory), which $\textit{implies}$ that the well-known "circle of fifths" emerges in learned chord embeddings (machine learning). (iii) We explain and reproduce a recently discovered spherical representation of celestial objects in large language models (LLMs) as a spherical-harmonic embedding derived from our theory. Our results demonstrate that the geometry of learned representations is often a consequence of the statistical symmetry of underlying data.

Problem

• word embeddings
• representation geometry
• group invariance
• co-occurrence statistics
• language models

Innovation

• Group-Invariant Statistics
• Irreducible Representations
• Embedding Geometry
• Harmonic Analysis
• Representation Learning