Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability

2026-06-03

AI Summary

This work investigates the interplay among parameters, data, and representations in deep neural networks, revealing that a multitude of approximately equivalent solutions persist even in structurally asymmetric models. To formalize this phenomenon, the authors introduce the theoretical framework of “effective function classes,” which characterizes the functions realizable by neurons over their input support along with their associated norm costs. They further propose the notion of “neuron identifiability” to rigorously capture effective symmetry breaking. This approach enables representation alignment without requiring prior correspondence and establishes sufficient conditions for the existence of low-loss linear interpolation paths between solutions. The study elucidates how effective function classes structurally shape the loss landscape and offers novel insights into the redundancy and generalization capabilities of deep models.

Abstract

Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged. Despite growing attention to parameter symmetries, the exact interplay between parameters, data, and representations remains underexplored. To investigate this, we develop a theoretical framework of effective function classes, i.e., the set of functions a neuron can realize on its input support, and the norm cost of realizing them. We then formalize effective symmetry breaking via neuron identifiability across independent training runs. Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in structurally asymmetric models. We further show that neuron identifiability enables representation merging without prior alignment, and characterize when such merging admits a linear low-loss path. These findings highlight the role of effective function classes in affecting the loss landscape.

Problem

• parameter symmetries
• linear mode connectivity
• neuron identifiability
• effective function classes
• loss landscape

Innovation

• effective function classes
• neuron identifiability
• linear mode connectivity
• symmetry breaking
• representation merging