ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

2026-07-22

AI Summary

This work addresses the quadratic complexity of Transformer attention, which hinders scalability to long sequences. The authors propose ELSAA, a novel method that jointly models sparse and low-rank structures directly within the attention operator: a sparse branch captures local high-similarity interactions, while a low-rank branch encodes global diffuse dependencies. A denominator-aware fusion mechanism dynamically balances the contributions of both branches. Notably, ELSAA avoids explicit computation of the full attention matrix and does not decompose projection weights. This design significantly reduces computational overhead while preserving fine-grained token-level interactions and effective long-range context integration, enabling efficient and scalable training on long sequences.

Abstract

The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths. Existing efficient attention methods usually reduce this bottleneck by either imposing sparsity, so that each query attends to only a small subset of keys, or by using low-rank/kernel sketches, so that global interactions are compressed into a lower-dimensional representation. We propose \emph{ELSAA}, an efficient low-rank and sparse approximation of attention. Importantly, ELSAA does \emph{not} decompose the learned projection or output matrices of the Transformer into sparse and low-rank factors. Instead, after dense projections produce $Q,K,V$, ELSAA approximates the induced attention score operator itself: a sparse branch captures selected high-similarity interactions, while a low-rank branch summarizes diffuse global interactions. Since the two branches can be normalized over supports with very different denominator mass, ELSAA introduces a denominator-aware fusion term that scales the sparse branch according to its estimated attention mass relative to the low-rank branch. This gives a practical framework for constructing low-rank and sparse attention outputs without materializing the full quadratic score matrix, aiming to enable longer-context training while preserving both sharp token-level interactions and broad contextual mixing.

Problem

• attention approximation
• long-context training
• quadratic complexity
• sparse attention
• low-rank approximation

Innovation

• low-rank approximation
• sparse attention
• attention fusion
• efficient Transformers
• long-context training