Memory Capacity of Nonlinear Recurrent Networks: Is it Informative?

2025-02-07

AI Summary

This paper addresses the fundamental question of whether the memory capacity (MC) of nonlinear recurrent neural networks (RNNs) reliably characterizes their ability to process stochastic signals. Leveraging a theoretical framework that integrates Kalman controllability matrix analysis, random matrix theory, and nonlinear dynamical systems modeling—and under a probabilistic weight distribution assumption—we rigorously derive the dependence of MC on input scaling. Our key contribution is the first formal proof that MC for nonlinear RNNs lacks robustness: its value depends solely on input scale and can be arbitrarily tuned within practical ranges, thus failing to reflect genuine temporal modeling capability. This result invalidates MC as a universal performance metric, demonstrating its lack of discriminative power both in linear and nonlinear regimes. Consequently, we fundamentally challenge the theoretical validity and practical utility of the prevailing MC definition.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

The total memory capacity (MC) of linear recurrent neural networks (RNNs) has been proven to be equal to the rank of the corresponding Kalman controllability matrix, and it is almost surely maximal for connectivity and input weight matrices drawn from regular distributions. This fact questions the usefulness of this metric in distinguishing the performance of linear RNNs in the processing of stochastic signals. This note shows that the MC of random nonlinear RNNs yields arbitrary values within established upper and lower bounds depending just on the input process scale. This confirms that the existing definition of MC in linear and nonlinear cases has no practical value.

Problem

• Assesses memory capacity in nonlinear RNNs
• Questions utility of current MC metrics
• Shows MC varies with input scale

Innovation

• Nonlinear RNNs memory capacity analysis
• Input process scale determines MC bounds
• MC lacks practical value in RNNs