Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation

2026-07-20

AI Summary

This work addresses numerical instability, gradient starvation, and phase anisotropy distortion inherent in conventional root-mean-square normalization—issues stemming from scalar squared accumulation—by proposing MRSNorm. The method structures channels into paired two-dimensional phasors, computes local L2 magnitudes under phase manifold constraints, and aggregates them globally via L1 averaging. It inverts the traditional scaling paradigm to preserve conformal invariance, shares affine weights to halve parameter count, and embeds a gradient-clipping mechanism grounded in the Pythagorean identity to ensure gradient homogeneity. Evaluated on CIFAR-100 with ResNet, MRSNorm maintains optimization stability even with half the parameters and effectively suppresses numerical explosion under extreme hyperparameter settings.

Abstract

While Root Mean Square Normalization has become the de facto standard for accelerating modern sequence models, its reliance on the quadratic accumulation of independent scalars ($\sum x^2$) inherently triggers outlier-induced numerical instability, gradient starvation, and anisotropic phase distortion. We introduce Mean Root Square Normalization (MRSNorm). By structurally pairing channels into 2D phasors, MRSNorm mathematically inverts the traditional scaling paradigm: it computes the localized $L_2$ magnitudes (Root Square) before aggregating them via a global $L_1$ average (Mean). This operational inversion strictly constrains activations to a phasor manifold, preserving conformal invariance. By sharing a single affine weight across phasor components, MRSNorm halves the total number of learnable parameters, proving that unconstrained spatial scaling in standard norms is a harmful redundancy. We analytically demonstrate that this geometric constraint yields a built-in, trigonometric gradient clipper governed by the Pythagorean identity, unconditionally equalizing the local gradient norm to ensure Gradient Homogeneity. Empirical evaluations on a ResNet with CIFAR-100 show that despite halved parameters, MRSNorm provides critical structural stability under rigorous stress tests. Under extreme hyperparameter settings where standard normalizations suffer from gradient divergence, MRSNorm successfully prevents numerical explosion and secures stable optimization trajectories. Our findings propose a fundamental paradigm shift toward phasor-based deep representation learning. The implementation of MRSNorm is available at Appendix C.

Problem

• numerical instability
• gradient starvation
• phase distortion
• anisotropy
• outlier-induced

Innovation

• Phasor Attention
• Mean Root Square Normalization
• Gradient Homogeneity
• Phase Manifold
• Conformal Invariance