Training-Free Hashing-Based Attention via Binary Principal Components

2026-08-04

AI Summary

This work addresses the computational inefficiency of long-context large language models during decoding, where self-attention incurs substantial overhead due to repeatedly processing an ever-growing key-value cache. To overcome this, the authors propose BinaryPC, a training-free, data-aware hashing-based sparse attention mechanism that introduces binary principal component analysis into attention computation for the first time. By constructing compact binary hash codes and hash functions that preserve intrinsic data structure, BinaryPC enables highly efficient approximate attention. Evaluated across multiple models and long-context benchmarks, BinaryPC achieves accuracy on par with full attention while delivering a 3.56× higher inference throughput than FlashAttention, significantly outperforming existing sparse and hashing-based baselines.

Abstract

Long-context large language models (LLMs) are increasingly deployed in real-world applications, yet self-attention remains a major efficiency bottleneck -- especially during decoding -- due to the necessity of repeatedly processing ever-growing key-value (KV) caches. Existing sparse attention reduce computation by attending to fewer KV pairs, but often suffer from substantial accuracy degradation, require additional training, or rely on expensive hashing. In this work, we present BinaryPC, a training-free, data-aware hashing-based sparse attention for long-context LLMs. BinaryPC constructs compact binary hash codes and corresponding hash function by computing binary principal components of data. Unlike Locality-Sensitive Hashing (LSH) with data-independent random projections or learned non-linear hashing methods, BinaryPC constructs binary codes that explicitly preserve the structural information of data without requiring gradient-based training. Comprehensive experiments across multiple model families and long-context benchmarks show that BinaryPC preserves accuracy relative to full attention while achieving superior performance among sparse and hashing-based baselines. On modern GPUs, BinaryPC improves end-to-end decoding throughput by 3.56$\times$ over the FlashAttention kernel. Our code is available at https://github.com/yudaohai666/BPC.

Problem

• long-context LLMs
• self-attention bottleneck
• sparse attention
• KV cache
• hashing-based attention

Innovation

• Binary Principal Components
• Training-Free Hashing
• Sparse Attention
• Long-Context LLMs
• Data-Aware Hashing