Dynamics and Representation Structure of Local Approximations to Gradient-Based Learning in Linear Recurrent Neural Networks

2026-05-29

AI Summary

This study investigates the learning dynamics of local gradient approximation algorithms—such as Random Feedback Local Online (RFLO) and truncated Backpropagation Through Time (tBPTT)—in brain-inspired recurrent neural networks constrained by spatiotemporal locality, and elucidates their fundamental differences from standard BPTT. Leveraging dynamical systems theory, orthogonal mode decomposition, and linear RNN modeling, the work provides the first dynamical-systems-based characterization of the steady-state solutions, stability, and convergence behavior of such local learning rules. It reveals that RFLO solutions are confined to low-rank perturbations of the initial parameters, exhibiting an intrinsic low-rank update structure, thereby exposing a fundamental limitation imposed by locality constraints on the network’s representational capacity. These findings establish a new theoretical foundation for neuromorphic computing and biologically plausible plasticity models.

Abstract

Biological and neuromorphic recurrent neural networks (RNNs) are subject to spatial and temporal locality constraints on the information that can plausibly be used during learning. A common strategy to satisfy these constraints is to modify gradient descent by neglecting non-local terms to varying degrees, as in random feedback local online (RFLO) learning and truncated backpropagation through time (tBPTT). However, the learning dynamics of these algorithms, and how they compare with BPTT, remain poorly understood. We apply dynamical systems theory to data-aligned linear RNNs -- whose dynamics can be separated into orthogonal modes -- to compare stationary solutions, stability properties, and convergence rates, finding qualitatively distinct behaviour for RFLO versus BPTT and one-step tBPTT. We further observe that the solutions learned by RFLO are restricted to low-rank perturbations of initial parameters, a result which holds beyond the data-aligned setting. Our work provides analytical insight into how locality constraints shape learning dynamics, with implications for neuroscientific models of learning and alternative optimization approaches for RNNs.

Problem

• locality constraints
• learning dynamics
• recurrent neural networks
• gradient-based learning
• BPTT

Innovation

• local learning
• recurrent neural networks
• gradient approximation
• dynamical systems
• low-rank perturbation