On Parameter Symmetries and Conservation Laws in Gradient Flow

2026-09-28

AI Summary

This study addresses the unclear relationship and applicable conditions between parameter symmetries and conservation laws in gradient flow. By integrating differential geometry with an analogy to Noether’s theorem, it constructs a unified geometric framework to elucidate their intrinsic connection. The work introduces the novel concept of “combinatorial identifiability” and establishes a general inheritance principle for conservation laws in multi-layer networks. Furthermore, it successfully characterizes the symmetry and conserved quantity structures in multi-head attention, polynomial, and deep linear networks. Overall, this research provides a systematic geometric perspective for the theoretical analysis of deep learning architectures.

Abstract

Parameter space symmetries and conservation laws play an important role in understanding the loss landscapes and implicit biases of neural networks. Inspired by Noether's theorem in physics, prior works have sought to derive conservation laws under gradient flow from parameter symmetries, but the scope and limitations of this connection remain unclear. We develop a unified geometric framework that clarifies the precise relationship between the two notions, including the conditions under which symmetries correspond to conservation laws. We introduce a notion of compositional identifiability and use it to establish a general inheritance principle for complete characterizations of symmetries and conservation laws in multilayer networks. We apply the framework to multi-head and grouped-query attention, polynomial neural networks, and square deep linear networks.

Problem

• parameter symmetries
• conservation laws
• gradient flow
• neural networks
• Noether's theorem

Innovation

• parameter symmetries
• conservation laws
• gradient flow
• compositional identifiability
• geometric framework