Approximate Speculative Decoding

2026-08-04

AI Summary

Standard speculative decoding discards all subsequently computed tokens upon the first token’s verification failure, leading to significant computational waste. This work proposes a training-free approximate verification mechanism that dynamically controls the maximum acceptable prefix length by leveraging target-logit regret gating, block-level anomaly upper bounds, and request-level regret budgets, thereby reusing greedy suffixes under bounded local deviations. The method requires neither a new draft model nor fine-tuning and strictly reduces to standard greedy verification when the regret budget is zero. Experiments demonstrate an average throughput improvement of 7.78% (up to 15.26%) on Qwen3-14B with DSpark-14B, and a 10%–16% increase in verification acceptance rates for DeepSeek-V4-Flash paired with DSpark on GSM8K and MATH-500 benchmarks.

Abstract

Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel. Under standard greedy verification, decoding stops at the first draft token that differs from the target argmax, discarding the remaining target-scored suffix. Although accepting such a mismatch changes the decoding trajectory, it can make a contiguous suffix reusable when its tokens remain target-greedy under the realized prefix. In this paper, we introduce \textbf{Approximate Speculative Decoding (ASD)}, a training-free verifier that replaces binary first-mismatch truncation with budgeted longest-prefix selection. ASD accepts selected mismatches subject to a local target-logit regret gate, a per-block exception cap, and a persistent request-level regret budget, then reuses the contiguous target-greedy suffix without additional approximate decisions or target-model forward passes. ASD requires neither a new draft model nor fine-tuning, and exactly reduces to standard greedy verification when the budget is zero. Experiments show that ASD improves fixed-workload throughput by $3.05\%$--$15.26\%$ over matched strict verification and averages a $7.78\%$ gain across seven Qwen3-14B + DSpark-14B tasks. On DeepSeek-V4-Flash (284B) with DSpark it also raises verifier-side acceptance by roughly $10\%$--$16\%$ on GSM8K and MATH-500 in an FP4-to-FP8 compatibility setting. The source code is publicly available at: https://github.com/Kissmetothemoon/ASD

Problem

• speculative decoding
• autoregressive generation
• greedy verification
• token acceptance
• decoding efficiency

Innovation

• Approximate Speculative Decoding
• speculative decoding
• training-free verification
• logit regret budget
• autoregressive generation