TiVaT: A Transformer with a Single Unified Mechanism for Capturing Asynchronous Dependencies in Multivariate Time Series Forecasting

2024-10-02

AI Summary

In multivariate time series forecasting, existing models struggle to capture complex asynchronous dependencies—such as lead-lag relationships—among variables, limiting predictive accuracy. To address this, we propose TiVaT, a unified time- and variable-aware joint modeling framework. TiVaT introduces a novel Joint-Axis self-attention mechanism that simultaneously models dynamic interactions along both the temporal and variable axes within a single module. It further incorporates a distance-aware 2D joint sampling strategy and learnable time- and variable-dependent positional embeddings, breaking away from conventional channel-separate paradigms. Extensive experiments on multiple benchmark datasets demonstrate that TiVaT consistently outperforms state-of-the-art methods. Notably, under strongly asynchronous scenarios, it achieves an average 12.7% reduction in prediction error, validating its effectiveness and generalizability in modeling asynchronous dependencies.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Computer Vision: Multi-modal Vision
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Multivariate time series (MTS) forecasting is vital across various domains but remains challenging due to the need to simultaneously model temporal and inter-variate dependencies. Existing channel-dependent models, where Transformer-based models dominate, process these dependencies separately, limiting their capacity to capture complex interactions such as lead-lag dynamics. To address this issue, we propose TiVaT (Time-variate Transformer), a novel architecture incorporating a single unified module, a Joint-Axis (JA) attention module, that concurrently processes temporal and variate modeling. The JA attention module dynamically selects relevant features to particularly capture asynchronous interactions. In addition, we introduce distance-aware time-variate sampling in the JA attention, a novel mechanism that extracts significant patterns through a learned 2D embedding space while reducing noise. Extensive experiments demonstrate TiVaT's overall performance across diverse datasets, particularly excelling in scenarios with intricate asynchronous dependencies.

Problem

• Multivariate Time Series Forecasting
• Asynchronous Relationships
• Leading-Lagging Variables

Innovation

• TiVaT Model
• Joint Axis Attention Module
• Distance-aware Temporal-Variable Sampling