equivariant networks

This paper addresses the unified modeling of symmetries in machine learning. It proposes a framework grounded in differential geometry and convex optimization to (1) enforce known symmetries, (2) automatically discover unknown symmetries in models or data, and (3) actively induce symmetry breaking via user-specified candidate groups. The core contribution is the first formulation of symmetry imposition and discovery as dual linear-algebraic tasks, leveraging the Lie derivative to characterize fiberwise linear Lie group actions on vector bundles, and employing nuclear-norm relaxation to construct convex regularization terms. The method is broadly applicable to neural networks, dynamical system discovery, basis-function regression, and neural operators. Empirically, it significantly improves generalization performance and parameter efficiency—particularly in low-data regimes—while preserving geometric structure and interpretability.

This paper investigates whether deep networks under standard supervised training can autonomously learn unencoded symmetries—such as rotation invariance—from partially observed cyclic group-symmetric data, under realistic class-level symmetry heterogeneity (where some classes exhibit full cyclic transformations while others only subsets). Method: We develop the first neural-kernel symmetry learning theory, grounded in infinite-width NTK analysis and group representation-theoretic Fourier analysis. Contribution/Results: The theory reveals that generalization to unseen symmetries hinges on the “overwhelming dominance” of local data structure over symmetry structure within the kernel-induced feature space, and yields a verifiable signal-to-noise criterion in the frequency domain. Our analysis precisely reproduces empirical failures of MLPs, CNNs, and ViTs on rotationally augmented MNIST subsets, and rigorously proves that conventional supervised training cannot acquire symmetries absent from the architectural prior.

Equivariant networks strictly preserve input symmetries, rendering them ill-suited for generative tasks requiring *active symmetry breaking*—e.g., reconstructing asymmetric structures from highly symmetric latent representations. To address this, we establish the first necessary and sufficient representation theorem for equivariant conditional distributions and propose SymPE: a method that achieves *controllable symmetry breaking* via learnable stochastic normalized positional encodings, while preserving the group-equivariant inductive bias. SymPE unifies probabilistic symmetry breaking, positional encoding, and equivariant graph neural networks, and naturally integrates with diffusion-based generative frameworks. Empirically, it significantly improves performance on graph diffusion modeling, graph autoencoding, and lattice spin system generation. Theoretically, we prove that SymPE’s generalization bound is strictly superior to that of conventional equivariant networks.

This work investigates how task-relevant symmetries—exact or approximate equivariance—affect the generalization of deep learning models, particularly under symmetry mismatch between model and data. Method: We develop the first generalization bound that does not assume group structure, rigorously quantifying the interplay between model equivariance error and data equivariance error. Our approach integrates probabilistic generalization theory, function approximation theory, and symmetry metrics, accommodating non-group, non-exact, and non-global equivariance settings. Contributions/Results: We establish that precise modeling of task symmetries significantly improves generalization. We formally characterize the optimal error trade-off under approximate or local equivariance when model and data symmetries are misaligned. Furthermore, we derive an “error alignment” principle—a concrete, actionable theoretical guideline for designing robust equivariant models—thereby bridging abstract symmetry considerations with practical architectural design.

This work investigates the learning dynamics of overparameterized neural networks on group-symmetric data. We develop a mean-field theoretical framework based on the action of a compact group (G), unifying the asymptotic training behaviors of three symmetry-exploiting techniques: data augmentation (DA), feature averaging (FA), and equivariant architectures (EA). We introduce the novel concepts of “weakly” and “strongly invariant distributions,” and prove that the mean-field dynamics in the infinite-width limit automatically preserve equivariance—overcoming the equivariance-breaking issue prevalent in finite-width networks. Theoretically, we show that all three methods converge to the same minimum-risk trajectory under symmetric data. Empirically, we demonstrate significantly improved accuracy of dynamical predictions in the large-width regime. Furthermore, we propose a heuristic method for learning equivariant subspaces, enabling automatic discovery of symmetry structures from data.

Existing equivariant networks typically support only fixed symmetry groups, limiting their ability to flexibly handle multimodal data with diverse symmetries. This work proposes the ASEN model, which achieves simultaneous equivariance to multiple permutation subgroups within a single architecture by incorporating symmetry-breaking auxiliary input features and leveraging an approximate symmetry-breaking mechanism together with an efficient 2-closure fast algorithm. Built upon a fully permutation-equivariant basis model and employing equivariant MLP emulation techniques, ASEN overcomes the rigidity of conventional equivariant networks. Experiments demonstrate that ASEN outperforms both specialized equivariant models and non-equivariant baselines across tasks involving graph and image symmetry selection, as well as sequence-based multitask and transfer learning scenarios.

This work investigates how Hopfield networks implicitly learn invariant representations of graph isomorphism classes from few random graph samples. Methodologically, we first prove that any graph isomorphism class can be embedded into a three-dimensional invariant subspace under the action of the permutation group; we then introduce Minimum Energy Flow (MEF) gradient descent, revealing its implicit bias toward norm efficiency, and derive a polynomial upper bound on sample complexity based on this bias. Theoretically, network parameters asymptotically converge to this invariant subspace as sample size increases; empirically, the mechanism enables efficient isomorphism class inference and strong generalization. Our core contributions are: (i) establishing the first theoretical framework for implicit invariant learning in Hopfield networks; (ii) uncovering an intrinsic unification between implicit bias—specifically norm-efficient optimization—and group invariance; and (iii) providing a novel principle for few-shot learning on graph-structured data.

This work investigates how to automatically uncover low-dimensional constraint structures induced by symmetries and conservation laws from high-dimensional physical data in the absence of explicit prior knowledge. To this end, we propose an unsupervised representation learning framework based on variational autoencoders that eschews conventional designs relying on explicit symmetry embeddings. Instead, our approach leverages the information bottleneck principle to drive the latent space to self-organize and reveal the dimensionality reduction inherent to underlying symmetries. Evaluated on geometric systems and particle physics datasets, the method successfully recovers theoretically expected symmetry structures and systematically delineates the theoretical limits and practical challenges of symmetry inference under minimal inductive bias.

Existing equivariant neural fields struggle to handle inconsistent group actions on heterogeneous product spaces. This work proposes an isotropy subgroup reduction framework that establishes an orbit equivalence $(X \times M)/G \cong X/H$, thereby transforming the learning of $G$-invariant functions over the product space into learning $H$-invariant functions solely on $X$, where $H$ is the isotropy subgroup. By circumventing the stringent structural constraints on group actions imposed by prior methods, this approach significantly enhances modeling flexibility while preserving expressive capacity. It achieves, for the first time, a unified equivariant modeling framework applicable to arbitrary group actions and homogeneous configuration spaces.

This work addresses the underexplored role of optimizers in training equivariant and geometric neural networks, which, despite their ability to encode geometric symmetries, often underperform unconstrained models due to optimization challenges. For the first time, we systematically compare the Muon and Adam optimizers across a range of equivariant architectures, employing Hessian curvature estimation, loss landscape visualization, and spectral analysis of weights—including stable and effective ranks—to reveal how optimizer choice profoundly shapes training dynamics and representational properties. On ModelNet40 point cloud classification and molecular tasks, Muon consistently outperforms Adam across all architectures, yielding models with smoother loss landscapes, higher curvature, and higher-rank weight matrices and intermediate features, thereby highlighting the critical interplay between optimizer design and geometric inductive biases.