Discovering Symmetries in Neural Network Parameter Spaces

2026-09-27

AI Summary

This study addresses the lack of systematic methods for identifying symmetries in neural network parameter spaces. To this end, it proposes an automated framework grounded in infinitesimal conditions that formalizes data-dependent symmetries and jointly learns generators alongside action mappings, thereby establishing a mechanism to extrapolate symmetry discovery from subnetworks to full models. The proposed approach successfully identifies both known and previously undiscovered parameter symmetries across diverse architectures, including pretrained Transformers. These contributions offer a novel analytical tool for elucidating the intrinsic structural properties of deep learning models.

Abstract

Parameter space symmetries are important for understanding neural networks'loss landscape, training dynamics, and generalization. However, systematically identifying these symmetries remains a challenge. In this paper, we formalize data-dependent parameter symmetries and characterize loss invariance and the group-action axioms through infinitesimal conditions, which provide objectives for jointly learning group generators and nonlinear action maps. Our framework systematically uncovers parameter symmetries, including previously unknown ones. To study larger networks, we establish conditions under which subnetwork symmetries extend to the full model. The same construction gives an explicit family of finite-batch symmetries, providing both analytical examples and a foundation for discovery through small subnetworks. Using the infinitesimal characterization and subnetwork construction, we implement a framework for automated discovery of parameter symmetries, and successfully uncovered symmetries in various architectures, including pretrained transformer models.

Problem

• parameter space symmetries
• neural networks
• loss landscape
• symmetry discovery

Innovation

• parameter space symmetries
• infinitesimal conditions
• subnetwork extension
• automated discovery
• loss invariance