MiniMax Sparse Attention

2026-06-11

AI Summary

This work addresses the challenge of deploying large language models with million-token contexts, where standard softmax attention incurs prohibitive computational costs. To overcome this, the authors propose a block-sparse attention mechanism built upon grouped-query attention (GQA). Their approach employs a lightweight indexing branch that independently selects top-k key-value blocks for each GQA group, enabling the main branch to perform exact sparse attention only over the chosen blocks. Innovatively integrating group-specific sparse retrieval with block-level efficient execution, they co-design an exponentiation-free top-k selection strategy and a sparse attention kernel based on outer products of key-value pairs, substantially improving tensor core utilization. Evaluated on a 109B-parameter multimodal model with 1M-token context, the method reduces attention computation by 28.4× and achieves 14.2× and 7.6× speedups in prefill and decode phases, respectively, on H800 GPUs, matching GQA’s performance.

Abstract

Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale. We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA). A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group, enabling group-specific sparse retrieval while maintaining efficient block-level execution; the Main Branch then performs exact block-sparse attention over only the selected blocks. Designed around a principle of simplicity and scalability, MSA is deliberately streamlined, making it straightforward to deploy efficiently across a broad range of GPUs. To translate sparsity into practical speedups, we co-design MSA with a GPU execution path that uses exp-free Top-k selection and KV-outer sparse attention to improve tensor-core utilization under block-granular access. On a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4x at 1M context. Paired with our co-designed kernel, MSA achieves 14.2x prefill and 7.6x decoding wall-clock speedups on H800. Our inference kernel is available at: https://github.com/MiniMax-AI/MSA. A production-grade natively multimodal model powered by MSA has been publicly released at: https://huggingface.co/MiniMaxAI/MiniMax-M3.

Problem

• ultra-long-context
• softmax attention
• quadratic cost
• sparse attention
• LLM deployment

Innovation

• Sparse Attention
• Grouped Query Attention
• Blockwise Sparsity
• GPU Kernel Co-design
• Ultra-long Context