A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

2026-07-30

AI Summary

This work addresses the inefficiency of speculative decoding in large language models when scaling to long contexts, which is hindered by the coupling among draft generation overhead, token acceptance rate, and speculation length. The authors propose SparseSpec-L, a training-free self-speculative framework that generates lightweight drafts through dynamically sparse yet recallable KV caches and reuses attention statistics from the verification phase as importance signals. Furthermore, an entropy-based online controller is introduced to dynamically optimize the speculation length at each decoding step. By uniquely integrating recallable sparse KV caching with adaptive length control, SparseSpec-L achieves significant end-to-end speedups across diverse models and long-context tasks—surpassing standard autoregressive decoding in peak performance—while strictly preserving the output distribution of the target model.

Abstract

Speculative decoding alleviates the memory-bandwidth bottleneck in large language model inference, but its acceleration is jointly constrained by drafting overhead, token acceptance, and speculation length. We present a unified efficiency analysis showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost. Guided by this analysis, we introduce SparseSpec-L, a training-free self-speculative decoding framework for long-context inference. SparseSpec-L generates lightweight drafts directly from the target model using a dynamically sparsified and recallable KV cache. It recycles per-head attention statistics produced during full-context verification as a no-extra-forward importance signal, allowing critical historical tokens to be recalled without permanently discarding the dense KV cache. An online entropy-based controller further selects the speculation length according to expected step-wise efficiency. Experiments across multiple long-context tasks and model scales show consistent end-to-end acceleration, with up to speedup over autoregressive decoding while preserving the target model's output distribution.

Problem

• speculative decoding
• memory-bandwidth bottleneck
• long-context inference
• token acceptance
• inference acceleration

Innovation

• self-speculative decoding
• training-free
• sparse KV cache
• attention statistics recycling
• entropy-based controller