🤖 AI Summary
This work addresses the limited dynamical expressivity of gated recurrent neural networks (RNNs) in reservoir computing, which often arises from suboptimal fixed-weight initialization. By leveraging random matrix theory and phase transition analysis, the authors derive a critical gain criterion applicable to various gated RNN architectures in the infinite-width limit. This criterion guides weight initialization such that the network operates precisely at the edge of chaos—the critical point between ordered and chaotic dynamics. Notably, it establishes the first direct link between the critical weight variance and peak performance in chaotic time series prediction tasks, accurately predicting the optimal initialization gain. The result provides a universal principle for the efficient design of gated RNNs, ensuring maximal computational capacity through principled initialization.
📝 Abstract
Proper weight initialization prior to training has historically been one of the key factors that helped kick off the deep learning revolution. Initialization is even more crucial in "reservoir computing", where the weights of a readout layer are learned linearly while the reservoir weights are fixed and largely determine the richness, stability and memory of the resulting dynamics. In the infinite-width limit it has been shown that meaningful initializations are those sitting at an effective critical point of the randomly initialized model. The phase transition is controlled by the weight variance $g^2$ and separates an ordered phase from a chaotic one where information progressively degrades. Here we derive a simple criterion to estimate the critical $g_c$ for a broad class of recurrent architectures and we show that it closely tracks the gain at which a gated-RNN reservoir achieves peak performance on a chaotic forecasting task. Finally, we argue that our criterion can serve as a design principle for future initialization schemes.