Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling

2026-10-06

AI Summary

This study addresses the quadratic scalability bottleneck of self-attention in modeling multidimensional long sequences by proposing TuBA. This method leverages low-rank tensor structures to project inputs into a compact Tucker core, performs multi-head attention within the core, and reconstructs representations back to the original space, enabling efficient global token mixing with sub-quadratic complexity. Its core innovation lies in introducing a Tucker bottleneck architecture alongside a multi-frame generation strategy, combining intra-core bidirectional interactions with cross-core causal attention for autoregressive scaling. Evaluated on video prediction and global weather forecasting tasks, TuBA reduces errors by up to 37.1% compared to full-rank baselines while decreasing computation by 85.1% and achieving a 4.27× speedup, demonstrating an exceptional trade-off between accuracy and efficiency.

Abstract

The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tensor structure for efficient global token mixing. TuBA projects hidden tensors into compact Tucker cores, performs multi-head self-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores. On video prediction and global weather forecasting, TuBA achieves favorable accuracy-efficiency trade-offs over standard and efficient attention and task-specific models. Compared to standard self-attention, TuBA reduces error and computation by up to 24.7% and 66.6% for video prediction and 37.1% and 85.1% for autoregressive weather forecasting, with speedups up to 4.27 times. Low-rank Tucker cores and multi-frame generation also outperform full-rank attention and frame-by-frame generation, respectively.

Problem

• self-attention
• multi-dimensional sequence modeling
• long sequences
• computational complexity
• scalability

Innovation

• Tucker bottleneck attention
• low-rank tensor
• subquadratic computation
• multi-dimensional sequence modeling
• autoregressive extension