Dropping the Anchor: Statistical Context Summarization for Distributed Systems via Pulsar Attention

2026-05-14

AI Summary

This work addresses the high computational cost of self-attention in long-sequence large language model inference, where existing distributed approaches rely on static anchor blocks that constrain both efficiency and performance. The authors propose Pulsar Attention, which eliminates static anchors and instead introduces a lightweight, content-aware attention sink prefix along with a cross-block dynamic summarization mechanism guided by a Max-IDF heuristic. This design achieves substantially improved computational efficiency and long-context comprehension while maintaining the same KV cache memory footprint. Compared to Star Attention, Pulsar reduces Phase 1 computation on a single GPU by up to 3.3×. On sequences of 128K tokens, it matches or exceeds the performance of dense attention across most tasks, with absolute gains of up to 4.7%.

Abstract

Inference with large language models (LLMs) on long sequences is computationally expensive due to the quadratic complexity of self-attention. Distributed blockwise methods such as Star Attention reduce this cost by sharding context across hosts, but rely on prepending a static, content-blind copy of the first block to every host. We propose Pulsar Attention, which replaces the static anchor with two lightweight, content-aware components: a small attention-sink prefix that stabilizes softmax, and compact cross-block summaries built via a Max-IDF heuristic that selects chunks containing globally rare tokens. This reduces the Phase 1 per-GPU FLOPs by up to 3.3$\times$ over Star Attention while retaining an identical KV cache footprint. On RULER and BABILong with Llama-3.1-8B, Pulsar Attention outperforms both Star Attention and dense attention at sequence lengths up to 128K tokens, with absolute gains of up to 4.7% over the dense baseline.

Problem

• long-context inference
• distributed attention
• quadratic complexity
• content-aware summarization
• LLM efficiency

Innovation

• Pulsar Attention
• distributed LLM inference
• context summarization
• Max-IDF heuristic
• attention sink