A Survey of Mamba

2024-08-02

arXiv.org

AI Summary

Transformers face inference bottlenecks on long sequences due to the quadratic computational complexity of self-attention. This paper presents a systematic survey of the emerging Mamba architecture—a linear-complexity sequence modeling paradigm grounded in state space models (SSMs). We establish the first comprehensive research landscape of Mamba, proposing a three-dimensional taxonomy—“architectural design–data adaptation–application scenarios”—to unify over one hundred cutting-edge works. We identify core technical pathways, including structured state propagation, hardware-aware scanning, and multimodal tokenization. Furthermore, we rigorously characterize Mamba’s theoretical advantages over Transformers in long-horizon time-series modeling, real-time inference, and lightweight deployment, while delineating its practical limitations. Finally, we articulate key open challenges and promising directions for future breakthroughs.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

As one of the most representative DL techniques, Transformer architecture has empowered numerous advanced models, especially the large language models (LLMs) that comprise billions of parameters, becoming a cornerstone in deep learning. Despite the impressive achievements, Transformers still face inherent limitations, particularly the time-consuming inference resulting from the quadratic computation complexity of attention calculation. Recently, a novel architecture named Mamba, drawing inspiration from classical state space models (SSMs), has emerged as a promising alternative for building foundation models, delivering comparable modeling abilities to Transformers while preserving near-linear scalability concerning sequence length. This has sparked an increasing number of studies actively exploring Mamba's potential to achieve impressive performance across diverse domains. Given such rapid evolution, there is a critical need for a systematic review that consolidates existing Mamba-empowered models, offering a comprehensive understanding of this emerging model architecture. In this survey, we therefore conduct an in-depth investigation of recent Mamba-associated studies, covering three main aspects: the advancements of Mamba-based models, the techniques of adapting Mamba to diverse data, and the applications where Mamba can excel. Specifically, we first review the foundational knowledge of various representative deep learning models and the details of Mamba-1&2 as preliminaries. Then, to showcase the significance of Mamba for AI, we comprehensively review the related studies focusing on Mamba models' architecture design, data adaptability, and applications. Finally, we present a discussion of current limitations and explore various promising research directions to provide deeper insights for future investigations.

Problem

• Transformers have quadratic complexity in attention computation
• Mamba offers linear scalability for sequence length processing
• Survey reviews Mamba's architecture, adaptability, and applications

Innovation

• Mamba architecture replaces Transformers efficiently
• State space models enable near-linear scalability
• Adapts Mamba to diverse data types