Parameter symmetries determine representational geometry in overparameterized nonlinear networks

2026-09-30

AI Summary

This study addresses the geometric degeneration of representations in overparameterized networks caused by parameter symmetries, which hinders the inference of computational functions from learned representations. Through group-theoretic analysis and feature-level transformation modeling, this work reveals that such symmetries are driven exclusively by three primitives: addition, duplication, and scaling. It proposes a closed-form decomposition method for representation geometry and designs a layer-selection rule algorithm to eliminate degeneration effects. Furthermore, this research establishes the boundary conditions under which computational functions can be inferred from representations, yielding an identifiable weighted feature geometry. By systematically elucidating the constraint mechanisms among parameter symmetries, representation geometry, and computational functions, this project provides a rigorous theoretical framework for understanding how representational structure governs functional expressivity in deep neural networks.

Abstract

Representations are routinely used across machine learning, psychology, and neuroscience to draw inferences about the computations of biological and artificial systems. Such inferences presume a meaningful link between representational geometry and the computation being performed. For artificial neural networks, however, the extent to which function constrains representation remains unclear. One key obstacle is that these networks admit parameter symmetries: changes in parameterization that preserve function exactly while reshaping representational geometry. Here, we show that a broad class of parameter symmetries acts on representations through just three primitive feature transformations: addition, duplication, and scaling. This feature-level characterization yields a closed-form decomposition of representational geometry into essential and auxiliary components, which makes precise how degeneracy in representational geometry can grow with overparameterization even when function is held fixed. Finally, we show that implementation-level selection rules can resolve this degeneracy, yielding identifiable geometries in which features are weighted according to their contributions to the network's function. Together, our results delineate when representations can support inferences about computation, and when they cannot.

Problem

• representational geometry
• parameter symmetries
• overparameterized networks
• degeneracy
• artificial neural networks

Innovation

• parameter symmetries
• representational geometry
• overparameterization
• feature transformations
• identifiable geometries