The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends

2026-09-30

AI Summary

This study addresses the quadratic computational overhead of LLM self-attention and long-context KV cache bottlenecks that constrain model scaling. It proposes unifying efficient sequence architectures as internal contextual memory within models. Drawing on 59 representative works, this paper reconstructs their evolutionary trajectory, introduces a multidimensional memory routing hypothesis, and establishes a five-dimensional analytical framework encompassing explicit compression and sparse access to systematically examine trade-offs among mechanisms. The contributions reveal network depth as a critical dimension for memory construction, emphasizing coordination across heterogeneous layers rather than isolated operator design. Furthermore, this work demonstrates that the core of efficient architectures lies in memory organization, lifecycle management, and selective retrieval, while identifying an emerging trend toward interface convergence across different mechanisms.

Abstract

Self-attention gives LLMs fine-grained, query-dependent access to context, but dense token interactions incur quadratic prefill cost and a key--value cache growing with context length. Research thus spans explicit-memory compression, sparse access, recurrent state construction, structured state dynamics, and heterogeneous mechanism composition. This survey analyzes these developments as model-internal contextual memory. We introduce a five-dimensional lens---Memory Representation, Memory Update, Access, Readout, and Integration---describing what is represented, how it changes, what is query-eligible, how it is read, and how readouts form outputs. This lens compares overlapping research lines without imposing one computational model. We reconstruct mechanism-level developments and architectural adoption using 59 release-level records from 14 major model lineages and 11 high-performing open-weight endpoints. First, explicit-memory and recurrent-state methods retain distinct interfaces but increasingly control overlapping memory functions. Second, heterogeneous architectures increasingly coordinate across network depth: layer-wise composition distributes complementary memory processing across representational stages, while cross-layer reuse carries selected memory and routing artifacts forward. Depth thus becomes a dimension along which contextual memory is constructed and managed. Third, these developments motivate a stateful multidimensional memory-routing hypothesis: persistent memory is organized across temporal scope, network depth, substrate type, and representation granularity, while coordinated Sparse Write and Sparse Read determine what is maintained and what contributes to each query. Overall, efficient sequence architecture design increasingly concerns the organization, lifecycle, and selective use of contextual memory rather than an isolated Attention operator.

Problem

• Large Language Models
• Self-attention
• Quadratic complexity
• KV cache
• Contextual memory

Innovation

• Attention Mechanisms
• Contextual Memory
• Five-Dimensional Framework
• Heterogeneous Architectures
• Memory Routing