recurrent neural networks

Structured State Space Models (SSMs) face a fundamental trade-off between long-range dependency modeling and computational efficiency, limiting their broad adoption across NLP, speech, vision, and time-series domains. This paper provides the first systematic survey of SSMs—from theoretical foundations (continuous-time dynamics, HiPPO projections) to industrial variants (S4, Mamba, S5, Jamba)—unifying analysis of their linear-time complexity, memory-efficient parameterization, and hardware-aware inference acceleration. We identify selectivity mechanisms and low-rank structured matrices as key innovations enabling SSMs to emerge as the third major sequence modeling paradigm—alongside RNNs and Transformers—achieving near-Transformer accuracy on long-sequence tasks while reducing memory footprint by over 70% and significantly improving inference throughput. We further highlight critical open challenges: training instability, hybrid modeling strategies, and interpretability.

Large language models (LLMs) inherently struggle to capture continuous temporal dynamics and explicit inter-variable dependencies in time series analysis. Method: This paper systematically reviews the emerging “time-series-to-image + vision model” paradigm, proposing the first dual-dimensional taxonomy: (i) time-series image encoding strategies (e.g., Gramian Angular Field, Markov Transition Field) and (ii) vision-model adaptation architectures (e.g., Vision Transformers, multimodal alignment, feature-decoupled reconstruction). It rigorously defines key pre-/post-processing challenges, surveys over 100 works, and establishes a unified evaluation framework. Contribution/Results: Empirical results demonstrate that vision-based approaches consistently outperform pure sequence models—achieving average accuracy gains of 5–12% across anomaly detection, forecasting, and classification tasks—thereby offering a promising new direction for time-series modeling.

Traditional RNNs suffer from weak representational capacity of hidden states and difficulty in modeling long-range dependencies. Method: This paper proposes the Test-Time Training (TTT) layer paradigm, which treats RNN hidden states as trainable modules—e.g., linear models or two-layer MLPs—and performs self-supervised updates per token during inference, enabling online adaptation of hidden states. Crucially, this retains linear time complexity while overcoming the fixed-parameter limitation of conventional RNNs. Contribution/Results: Experiments on models ranging from 125M to 1.3B parameters show that TTT-Linear and TTT-MLP achieve consistently decreasing perplexity on ultra-long contexts (>16K tokens), significantly outperforming Mamba and matching Transformer performance. This work is the first to empirically validate that linear-complexity architectures can simultaneously achieve strong expressivity and online learning capability through adaptive hidden-state design.

To address the limitations of RNNs and LSTMs in multistep time-series forecasting—including insufficient capacity to model complex nonlinear patterns, poor interpretability, and low computational efficiency—this paper proposes the TKAN architecture. Its core innovation is the Recurrent Kolmogorov–Arnold Network (RKAN) layer, which integrates learnable spline-based activations from Kolmogorov–Arnold Networks (KANs) into a gated recurrent structure for the first time. This design unifies dynamic weight evolution with long-term dependency modeling while ensuring end-to-end differentiability and strong universal approximation capability grounded in the Kolmogorov–Arnold representation theorem. Extensive experiments demonstrate that TKAN achieves significantly higher prediction accuracy and faster inference speed than state-of-the-art MLP, RNN, and LSTM baselines, without sacrificing model interpretability.

To address the limitation of sLSTM—its poor short-term memory retention, which hinders direct application to long-horizon time series forecasting (TSF)—this paper proposes P-sLSTM, the first model to synergistically integrate sequence patching and channel independence. P-sLSTM enhances local pattern modeling via patch-based encoding, while leveraging channel-wise decoupling, exponential gating, and memory mixing to mitigate short-term memory degradation without compromising sLSTM’s inherent capacity for long-range dependency capture. Theoretically interpretable, structurally lightweight, and parameter-efficient, P-sLSTM achieves state-of-the-art performance across multiple standard long-horizon TSF benchmarks, reducing average prediction error by 12.7% compared to prior methods. Moreover, it outperforms mainstream Transformer- and RNN-based variants in inference speed, demonstrating both accuracy and efficiency advantages.

Addressing challenges in long-term multivariate time series forecasting—including temporal dynamic modeling, cross-variable dependency capture, and multi-perspective reasoning—this paper proposes xLSTM-Mixer, a hybrid architecture. Methodologically: (1) it introduces the first fusion paradigm integrating extended LSTM (xLSTM) with Token-Mixer; (2) it designs a scalar memory mechanism to strengthen long-range dependency modeling; and (3) it incorporates cross-variable linear initialization alongside a dual-perspective (temporal and variable-dimensional) mixed output mechanism. The model is end-to-end differentiable and trained via standard backpropagation. Evaluated on benchmark datasets (ETT, Weather), xLSTM-Mixer consistently outperforms state-of-the-art models—including Informer, Autoformer, and PatchTST—with an average 12.3% reduction in MAE for long-horizon forecasts (96–192 steps). These results validate the effectiveness and renewed potential of enhanced recurrent architectures in modern time series forecasting.

This paper addresses poor model reproducibility and insufficient open-source implementations in time-series forecasting by proposing a lightweight, fully reproducible LSTM/GRU modeling paradigm. Methodologically, it constructs univariate sequence samples via sliding windows and evaluates performance using two metrics—RMSE and directional accuracy (DA)—on both synthetic activity data (Activities) and real-world financial data (BSE BANKEX). A key finding is that effective training requires only a single time series exhibiting repetitive patterns, without complex preprocessing or large-scale datasets. Experiments show that the proposed implementation significantly outperforms the “repeat last value” baseline for 1-step and 20-step predictions on Activities, while achieving comparable performance on BSE BANKEX. All code, datasets, and complete experimental configurations are publicly released to ensure full reproducibility and out-of-the-box usability.

Convolutional recurrent neural networks (ConvRNNs) suffer from inefficient sequential hidden-state updates, high parameter counts, and poor scalability. To address these limitations, we propose MinConvLSTM and MinConvGRU: lightweight convolutional RNNs that retain spatial inductive bias via convolutional operators while enabling fully parallel hidden-state computation through a log-domain prefix-sum mechanism. Inspired by xLSTM, we further incorporate exponential gating to enhance modeling capacity. Both models adopt a minimalist recurrent architecture, drastically reducing parameter count and computational complexity. Experiments on Navier–Stokes dynamics simulation and real-world potential field forecasting demonstrate that our models achieve faster training convergence, lower prediction error, and superior stability in closed-loop forecasting—outperforming standard ConvLSTM and ConvGRU baselines. The proposed architectures thus offer an effective trade-off among efficiency, accuracy, and generalization.

This work addresses the challenges of long-sequence modeling, where Transformers suffer from quadratic computational complexity and conventional RNNs are hindered by sequential computation that impedes parallelization. The authors propose PR-LSTM, a hierarchical recurrent architecture that transforms the nonlinear recurrence of hidden states into a parallelizable tree reduction process via a balanced computation tree. By uniquely integrating nonlinear gated state updates with logarithmic-depth parallelism, PR-LSTM overcomes the inherent serial bottleneck of RNNs while avoiding the high computational cost of attention mechanisms. Efficient hierarchical state composition is achieved through a fixed schedule based on parallel scan and a fused recursive gating module. Experiments demonstrate that PR-LSTM significantly outperforms standard RNNs, LSTMs, and Transformers on formal language tasks and exhibits strong length generalization capabilities.

Existing spatiotemporal forecasting models suffer significant performance degradation when input-label pairs exhibit spatiotemporal discrepancies—such as similar inputs leading to divergent futures. To address this, this work proposes ReLearner, a bidirectional learning framework that explicitly models and smooths the spatiotemporal residual between inputs and labels by incorporating label features. Grounded in a newly established spatiotemporal residual theorem, ReLearner integrates residual decoupling and smoothing modules to extend the conventional unidirectional prediction paradigm into a bidirectional learning process, enabling seamless integration into diverse spatiotemporal neural architectures. Extensive experiments across 11 real-world datasets and 14 backbone models demonstrate that ReLearner consistently and substantially improves forecasting accuracy, confirming its generality and effectiveness.

Conventional LSTM model selection relies on computationally expensive heuristics for capturing long-term dependencies and lacks a rigorous statistical foundation. Method: This paper introduces the first unified statistical framework for recurrent neural networks, integrating information criteria and shrinkage estimation—previously absent in time-series deep learning. It proposes a time-structured penalized likelihood function and a dynamic generalized thresholding method for latent state regularization, coupled with variational Bayesian inference and approximate marginal likelihood estimation for efficient model selection. Contribution/Results: Evaluated on multiple biomedical time-series datasets, the framework substantially mitigates overfitting, enhances prediction stability and generalization performance, and significantly reduces hyperparameter tuning cost—establishing a principled, statistically grounded alternative to heuristic LSTM model selection.

This work addresses the fragmentation in current linear recurrent neural network (LRNN) research, which is hindered by scattered implementations, strong framework dependencies, and a lack of standardized interfaces—impeding reproducibility, comparison, and extension. To bridge this gap, we introduce lrnnx, the first unified open-source library that integrates multiple modern LRNN architectures. Built on mainstream deep learning frameworks, lrnnx supports diverse parameterizations and discretization schemes without requiring custom CUDA kernels. The library offers a multi-level modular interface, ranging from low-level components to high-level models, substantially lowering the barrier to both usage and development. This design enables flexible control, facilitates fair benchmarking, and establishes a much-needed standardized toolkit for the LRNN community.