Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection

2026-07-23

AI Summary

This work proposes a parameter-free sparse attention mechanism that circumvents the high deployment costs of existing adaptive approaches, which often rely on learnable parameters, custom gradient estimators, or specialized CUDA kernels. The method introduces a novel, parameter-free signal for content selection by leveraging gzip compression ratios: it dynamically identifies information-rich segments of text through their compressibility and constructs sparse attention masks accordingly. This approach integrates seamlessly into standard Transformer architectures without architectural modifications. Evaluated on the PG-19 dataset with an 8K-token context window, the model achieves 1.71 bits per byte (BPB), substantially outperforming both dense and various sparse baselines. It also converges 3.3 times faster, with performance gains increasing as sequence length grows.

Abstract

Data-adaptive sparse attention masks substantially outperform fixed patterns (e.g., BigBird and Longformer) and can even exceed dense attention on long sequences. Existing adaptive approaches---including SBM-Transformer, Dynamic Mask Attention, and NSA---typically require additional learnable parameters, custom gradient estimators, or specialized CUDA kernels. We show that classical data compression provides an effective masking signal with \textbf{no additional parameters}. By computing per-block gzip compression ratios, we identify non-redundant content blocks and route long-range attention selectively through them. Intuitively, blocks that gzip cannot compress contain information not predictable from local repetition, making them natural long-range attention targets. Because the compression profile is input-dependent, the resulting sparse mask adapts dynamically to content without learned parameters, auxiliary losses, or custom kernels. On PG-19 byte-level language modeling at 92M parameters with 8K context, our method achieves 1.71 bits-per-byte (BPB), outperforming dense attention (2.89), BigBird (2.34), Longformer (3.21), and a reimplemented SBM-Transformer (3.38)---the only learned-mask baseline---by up to 1.67 BPB while adding no parameters. The advantage grows with sequence length, with the gap over BigBird widening from 0.05 BPB at 4K context to 0.63 BPB at 8K, while convergence is 3.3$\times$ faster.

Problem

• adaptive sparse attention
• parameter-free
• data compression
• long-context modeling
• attention masking

Innovation

• parameter-free
• adaptive sparse attention
• data compression
• gzip-based masking
• long-context modeling