Translation-Equivariance of Normalization Layers and Aliasing in Convolutional Neural Networks

2025-05-26

AI Summary

Existing normalization layers in convolutional neural networks lack a rigorous theoretical analysis of continuous translational equivariance. Method: We establish the first formal mathematical framework for equivariance in normalization layers by integrating group action theory with signal sampling analysis. We introduce a dual-equivariance definition—“discrete shift + continuous translation”—derive necessary and sufficient conditions for translational equivariance, and uncover its dimension-dependent mechanism. Contribution/Results: We prove that standard normalization schemes—including BatchNorm, LayerNorm, and InstanceNorm—are inherently non-equivariant due to cross-dimensional statistical aggregation over spatial or channel dimensions. Empirical validation on ResNet-18/ImageNet feature maps fully corroborates our theory, precisely delineating the equivariance boundaries of each normalization across dimensions. This work provides the first verifiable, physics-informed design principle for equivariant normalization layers in CNNs.

Technology Category

Computer Vision: Adversarial Attacks & RobustnessMachine Learning: Deep Learning TheoryNatural Language Processing: Safety and Robustness

Application Category

Security and Privacy: Data transparency and provenanceSocial Networks and Social Media: Fairness and bias in social network and social media analysisGraph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

The design of convolutional neural architectures that are exactly equivariant to continuous translations is an active field of research. It promises to benefit scientific computing, notably by making existing imaging systems more physically accurate. Most efforts focus on the design of downsampling/pooling layers, upsampling layers and activation functions, but little attention is dedicated to normalization layers. In this work, we present a novel theoretical framework for understanding the equivariance of normalization layers to discrete shifts and continuous translations. We also determine necessary and sufficient conditions for normalization layers to be equivariant in terms of the dimensions they operate on. Using real feature maps from ResNet-18 and ImageNet, we test those theoretical results empirically and find that they are consistent with our predictions.

Problem

• Study translation-equivariance in normalization layers of CNNs
• Establish conditions for equivariance in normalization layer dimensions
• Empirically validate theoretical findings using ResNet-18 and ImageNet

Innovation

• Theoretical framework for normalization layers equivariance
• Conditions for equivariance in normalization dimensions
• Empirical validation with ResNet-18 and ImageNet