Deep Residual Echo State Networks: exploring residual orthogonal connections in untrained Recurrent Neural Networks

2025-08-28

AI Summary

Traditional echo state networks (ESNs) exhibit limited capability in modeling long-term temporal dependencies. To address this, we propose DeepResESN—a deep residual ESN that, for the first time, integrates time-domain residual connections with orthogonal initialization in an untrained deep recurrent reservoir, yielding a hierarchical and dynamically stable residual recurrent architecture. Theoretical analysis guarantees Lyapunov stability of the reservoir dynamics, while two orthogonal residual configurations—random and fixed—are designed to enhance memory capacity. Extensive experiments on benchmark long-sequence tasks—including Mackey–Glass, Santa Fe, and NARMA—demonstrate that DeepResESN significantly outperforms classical ESNs and state-of-the-art deep reservoir models in terms of accuracy, robustness, and generalization. These results validate DeepResESN’s effectiveness for long-term temporal modeling, establishing a new paradigm for stable, scalable, and expressive reservoir computing.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Learning & Optimization for NLP
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Echo State Networks (ESNs) are a particular type of untrained Recurrent Neural Networks (RNNs) within the Reservoir Computing (RC) framework, popular for their fast and efficient learning. However, traditional ESNs often struggle with long-term information processing. In this paper, we introduce a novel class of deep untrained RNNs based on temporal residual connections, called Deep Residual Echo State Networks (DeepResESNs). We show that leveraging a hierarchy of untrained residual recurrent layers significantly boosts memory capacity and long-term temporal modeling. For the temporal residual connections, we consider different orthogonal configurations, including randomly generated and fixed-structure configurations, and we study their effect on network dynamics. A thorough mathematical analysis outlines necessary and sufficient conditions to ensure stable dynamics within DeepResESN. Our experiments on a variety of time series tasks showcase the advantages of the proposed approach over traditional shallow and deep RC.

Problem

• Enhancing long-term information processing in Echo State Networks
• Improving memory capacity and temporal modeling with residual connections
• Exploring orthogonal configurations for stable network dynamics

Innovation

• Deep untrained RNNs with temporal residual connections
• Orthogonal configurations for stable network dynamics
• Hierarchical residual layers boost memory capacity