Geometry of Linear Neural Networks: Equivariance and Invariance under Permutation Groups

2023-09-24

arXiv.org

AI Summary

This work systematically investigates equivariance and invariance of linear neural networks under arbitrary permutation group actions. Method: Leveraging algebraic geometry (determinantal varieties, invariant theory), representation theory of finite groups, and matrix analysis (Eckart–Young theorem), we derive exact geometric characterizations of linear equivariant and invariant function spaces. Contribution/Results: We provide the first complete algebraic characterization of the linear invariant function space under any permutation group—showing it is precisely parameterized by a single linear autoencoder satisfying cycle-decomposition constraints. We further reveal that the equivariant function space exhibits an intrinsic geometric structure: it is multiply connected and admits an irreducible decomposition. Our framework yields structurally simple, implementable parameterizations and corresponding optimization paths. Collectively, these results establish the first rigorous, general, and constructive theoretical foundation for symmetry-driven neural network design.

Abstract

The set of functions parameterized by a linear fully-connected neural network is a determinantal variety. We investigate the subvariety of functions that are equivariant or invariant under the action of a permutation group. Examples of such group actions are translations or $90^circ$ rotations on images. We describe such equivariant or invariant subvarieties as direct products of determinantal varieties, from which we deduce their dimension, degree, Euclidean distance degree, and their singularities. We fully characterize invariance for arbitrary permutation groups, and equivariance for cyclic groups. We draw conclusions for the parameterization and the design of equivariant and invariant linear networks in terms of sparsity and weight-sharing properties. We prove that all invariant linear functions can be parameterized by a single linear autoencoder with a weight-sharing property imposed by the cycle decomposition of the considered permutation. The space of rank-bounded equivariant functions has several irreducible components, so it can not be parameterized by a single network-but each irreducible component can. Finally, we show that minimizing the squared-error loss on our invariant or equivariant networks reduces to minimizing the Euclidean distance from determinantal varieties via the Eckart-Young theorem.

Problem

• Linear Neural Networks
• Symmetry Properties
• Efficient Network Design

Innovation

• Linear Neural Networks
• Invariant and Equivariant Properties
• Optimization Understanding