FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention

2026-06-08

AI Summary

This work addresses the GPU memory bottleneck in conventional large language models during ultra-long-context reasoning, which stems from maintaining a full key-value (KV) cache. The authors propose Lookahead Sparse Attention (LSA), a novel mechanism that employs a neural memory indexer to predict future critical context and retains only essential KV blocks. A decoupled training strategy enables independent optimization of the indexer without loading the backbone model. This approach pioneers an efficient “less-is-more” inference paradigm by integrating a dual-encoder architecture with KV cache compression. On LongBench-v2, it reduces physical KV cache usage to 13.5% of the baseline while improving average accuracy by 0.6%; under 500K-token contexts, cache overhead is cut by over 90% with no degradation in reasoning capability.

Abstract

Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving. In this report, we propose Lookahead Sparse Attention (LSA), a novel inference paradigm powered by a Neural Memory Indexer built upon the DeepSeek-V4 architecture. Rather than passively attending to all historical tokens, LSA proactively predicts future context demands and preserves only the query-critical KV chunks in the GPU memory. Crucially, we instantiate this architecture via a backbone-free decoupled training strategy. By formulating the indexer as a standard dual-encoder architecture, we train it independently using standard retrieval training frameworks without ever loading the massive backbone model into GPU memory. We demonstrate that this "less is more" paradigm significantly maximizes serving efficiency while acting as an effective attention denoiser in tasks that rely on long-term global memory. Across primary long-context evaluation suites (e.g., LongBench-v2, LongMemEval, and RULER), FM-DS-V4 compresses the average physical KV cache footprint down to merely 13.5% of the full-context baseline, while consistently preserving or slightly elevating downstream accuracy (+0.6% absolute margin on average). Crucially, at extreme 500K scales, FlashMemory suppresses the physical KV cache overhead by over 90% without destabilizing the backbone's core reasoning capacities.

Problem

• KV cache
• ultra-long context
• GPU memory bottleneck
• large language models
• inference efficiency

Innovation

• Lookahead Sparse Attention
• Neural Memory Indexer
• KV Cache Compression
• Decoupled Training
• Ultra-Long Context