🤖 AI Summary
Existing speculative decoding (SD) suffers from severe inference latency due to asynchronous execution between the draft and target models and fixed draft lengths, causing mutual waiting and limiting acceleration. This paper proposes PEARL, a novel framework that eliminates temporal coupling between draft and target models. PEARL introduces a synergistic pre- and post-verification mechanism to enable fully parallel draft generation and verification, and incorporates dynamic draft-length scheduling to adaptively determine the optimal speculation length per decoding step. These innovations decouple draft and target model execution both spatially and temporally. Evaluated on multiple text generation benchmarks, PEARL achieves a 4.43× speedup over autoregressive decoding and a 1.50× improvement over standard SD. The implementation is publicly available.
📝 Abstract
Speculative decoding (SD), where an extra draft model is employed to provide multiple draft tokens first, and then the original target model verifies these tokens in parallel, has shown great power for LLM inference acceleration. However, existing SD methods suffer from the mutual waiting problem, i.e., the target model gets stuck when the draft model is guessing tokens, and vice versa. This problem is directly incurred by the asynchronous execution of the draft model and the target model and is exacerbated due to the fixed draft length in speculative decoding. To address these challenges, we propose a conceptually simple, flexible, and general framework to boost speculative decoding, namely Parallel spEculative decoding with Adaptive dRaft Length (PEARL). Specifically, PEARL proposes pre-verify to verify the first draft token in advance during the drafting phase, and post-verify to generate more draft tokens during the verification phase. PEARL parallels the drafting phase and the verification phase via applying the two strategies, and achieves adaptive draft length for different scenarios, which effectively alleviates the mutual waiting problem. Experiments on various text generation benchmarks demonstrate the effectiveness of our PEARL, leading to a superior speed up performance up to 4.43$ imes$ and 1.50$ imes$, compared to auto-regressive decoding and vanilla speculative decoding, respectively. Our code is available at https://github.com/smart-lty/ParallelSpeculativeDecoding.