Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory

2026-08-03

AI Summary

Traditional continuous attractor networks are highly sensitive to parameter variations, while standard recurrent neural networks (RNNs) tend to fragment the state space into discrete attractors, hindering stable representation of continuous manifolds. Inspired by cortical divisive normalization mechanisms, this work proposes Recurrent Divisive Normalization Networks (RDNNs), which employ dynamic divisive operations to enable robust encoding of continuous working memory. Theoretical and empirical analyses demonstrate that divisive normalization is not only a biologically plausible mechanism but also a critical computational principle for preventing manifold fragmentation and preserving high-fidelity continuous representations. Within backpropagation through time (BPTT), it introduces activity-dependent gradient scaling. RDNNs converge to stable slow manifolds, effectively compressing the network’s effective rank and constraining dynamics to a low-dimensional subspace, significantly outperforming models relying solely on subtractive inhibition.

Abstract

The ability to robustly maintain and update continuous variables is a hallmark of working memory. While classical continuous attractor networks suffer from severe fine-tuning fragility, standard artificial recurrent neural networks (RNNs) like GRUs and LSTMs typically fail to stably learn continuous manifolds, instead shattering the state space into discretized point attractors. To bridge this gap, we draw inspiration from divisive normalization, a canonical neural computation widely observed across cortical circuits, and propose the Recurrent Divisive Normalization Network (RDNN), a minimal and algebraically isolated model of dynamic division. Through dynamical systems analysis on canonical working memory tasks, we demonstrate that this biophysical constraint allows the network to converge to robust, high-fidelity slow manifolds. Furthermore, we analyze the gradient dynamics of divisive normalization during Backpropagation Through Time (BPTT), showing that it introduces an activity-dependent local gradient scaling. This scaling dampens parameter updates in highly active regimes, which empirically aligns with a significant self-compression of the network's effective rank, confining the recurrent dynamics to a tight, low-dimensional subspace while avoiding the optimization pathologies associated with explicit low-rank factorization. Finally, ablations demonstrate that while subtractive inhibition can maintain static memories, divisive normalization is mathematically essential to prevent manifold shattering under time-varying inputs. Our findings identify divisive normalization not merely as a biological artifact, but as a critical computational mechanism for learning high-fidelity continuous representations.

Problem

• continuous working memory
• attractor networks
• manifold shattering
• recurrent neural networks
• divisive normalization

Innovation

• divisive normalization
• slow manifolds
• low-rank dynamics
• recurrent neural networks
• working memory