LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing

2026-08-02

AI Summary

This work addresses the system bottlenecks in deploying DeepSeek’s sparse attention, which suffers from O(L²) indexing overhead and non-contiguous memory accesses. To overcome these limitations, we propose LongCat, a sparse attention framework that co-designs hardware-aware algorithms through stream-aware indexing, cross-layer indexing (augmented with knowledge distillation), and a hierarchical coarse-to-fine scoring mechanism. This approach substantially reduces indexing computation and memory fragmentation. LongCat is the first to enable native training with million-token context lengths, achieving performance on par with full attention across models ranging from 69B to 560B parameters. The framework underpins the development of LongCat-2.0 (1.6T-A48B) and includes the open-sourced LongCat-Flash-Lite-Sparse model.

Abstract

DeepSeek Sparse Attention (DSA) enables efficient long-context modeling through its Lightning Indexer. However, practical deployment remains constrained by the indexer's expensive $O(L^2)$ scoring overhead and the hardware-inefficient, discontinuous memory-access patterns induced by its outputs. To address these system-level bottlenecks, we introduce LongCat Sparse Attention (LSA), a hardware-algorithm co-designed framework comprising three complementary and orthogonal strategies: (1) Streaming-Aware Indexing, which selectively converts scattered KV entries into hardware-aligned contiguous layouts to enable coalesced HBM access; (2) Cross-Layer Indexing, which amortizes indexing overhead by reusing the results produced by a single layer across consecutive layers, supported by cross-layer distillation; and (3) Hierarchical Indexing, which adopts a coarse-to-fine scoring scheme to progressively narrow the candidate set for each query, thereby substantially reducing indexing computation. Extensive scaling experiments, ranging from 69B-A3B to 560B-A27B models, demonstrate that LSA consistently achieves performance on par with full attention across both general-purpose and long-context benchmarks. Moreover, LSA supports native training with context lengths of up to one million tokens and underpins the development of LongCat-2.0 (1.6T-A48B). To facilitate further research, we also introduce and open-source LongCat-Flash-Lite-Sparse (69B-A3B), which integrates LSA into LongCat-Flash-Lite and incorporates an updated long-context training corpus.

Problem

• Sparse Attention
• Long-context Modeling
• Memory Access Patterns
• Indexing Overhead
• Hardware Efficiency

Innovation

• Streaming-Aware Indexing
• Cross-Layer Indexing
• Hierarchical Indexing
• Sparse Attention
• Hardware-Algorithm Co-design