Simplified Mamba with Disentangled Dependency Encoding for Long-Term Time Series Forecasting

2024-08-22

arXiv.org

AI Summary

Existing long-term time series forecasting (LTSF) models struggle to jointly model sequential and semantic dependencies along the time dimension, as well as cross-variable dependencies along the feature dimension—leading to error accumulation and degraded performance. Conventional state space models (SSMs) introduce redundant nonlinearity in semantically sparse time series, harming expressivity and efficiency. This paper proposes DecoMamba: the first LTSF framework to decouple dependency encoding, theoretically proving and empirically validating that removing Mamba’s nonlinear activation better aligns with sparse temporal semantics. We design a linear state space architecture integrating decoupled position-variable embeddings and learnable dependency gating, enabling low-interference joint modeling across time and variable dimensions. Evaluated on nine real-world datasets, DecoMamba achieves an average 12.7% reduction in MAE, 3.1× faster inference, and significantly improved generalization and robustness.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Data Mining & Knowledge Management: Mining of Spatial, Temporal or Spatio-Temporal Data

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent advances in deep learning have led to the development of numerous models for Long-term Time Series Forecasting (LTSF). However, most approaches still struggle to comprehensively capture reliable and informative dependencies inherent in time series data. In this paper, we identify and formally define three critical dependencies essential for improving forecasting accuracy: the order dependency and semantic dependency in the time dimension as well as cross-variate dependency in the variate dimension. Despite their significance, these dependencies are rarely considered holistically in existing models. Moreover, improper handling of these dependencies can introduce harmful noise that significantly impairs forecasting performance. To address these challenges, we explore the potential of Mamba for LTSF, highlighting its three key advantages to capture three dependencies, respectively. We further empirically observe that nonlinear activation functions used in vanilla Mamba are redundant for semantically sparse time series data. Therefore, we propose SAMBA, a Simplified Mamba with disentangled dependency encoding. Specifically, we first eliminate the nonlinearity of vanilla Mamba to make it more suitable for LTSF. Along this line, we propose a disentangled dependency encoding strategy to endow Mamba with efficient cross-variate dependency modeling capability while minimizing the interference between time and variate dimensions. We also provide rigorous theory as a justification for our design. Extensive experiments on nine real-world datasets demonstrate the effectiveness of SAMBA over state-of-the-art forecasting models.

Problem

• Capturing complex dependencies in time series data
• Reducing redundancy in State Space Models
• Improving forecasting accuracy via disentangled encoding

Innovation

• Simplifies SSMs by removing unnecessary nonlinearities
• Disentangles encoding for cross-variate dependencies
• Enhances SSMs for time series forecasting