From Chains to Trees: Parent-Conditioned Drafting for Semi-Autoregressive Speculative Decoding

2026-08-03

AI Summary

This work addresses a key limitation of existing semi-autoregressive draft models—such as DSpark—which employ linear generation structures wherein early token mismatches cause complete failure of subsequent predictions, thereby constraining the acceleration potential of large draft blocks. To overcome this, the authors propose a tree-based draft generation method that leverages a pretrained Markov head to independently score multiple child tokens for each parent node and prioritizes verification of higher-probability paths under a fixed verification budget. Notably, this approach requires no retraining or additional backbone forward passes, achieving improved acceleration solely through inference-time modifications. Evaluated on the Qwen3 model series across nine benchmarks, the method yields relative speedup gains of 3.1%–29.5% over DSpark; specifically, on GSM8K, Qwen3-4B increases its average accepted length from 9.41 to 11.16, raising the speedup ratio from 6.14× to 6.60×.

Abstract

Speculative decoding accelerates LLM inference only when drafted continuations survive target-model verification. Semi-autoregressive drafters such as DSpark predict an entire token block with one backbone forward and refine it with a lightweight Markov head. However, DSpark decodes this block as a single chain, so an early mismatch invalidates the remaining suffix and limits the benefit of large draft blocks. We show that the conditional structure already learned by DSpark can support multiple parent-consistent continuations without retraining or additional backbone passes. We introduce Parent-Conditioned Drafting Tree (PCTree), which uses the pretrained Markov head to score alternative children separately for each concrete parent and allocates a fixed verification budget to the most probable paths. This converts DSpark's linear draft into a tree while preserving its one-pass parallel backbone. Across Qwen3-{4B,8B,14B} and nine benchmarks, at $B{=}7$, measured speedup gains over autoregressive (AR) decoding, relative to matched DSpark, range from $3.1\%$ to $29.5\%$. On Qwen3-4B GSM8K at $B{=}16$, PCTree increases mean acceptance length from $9.41$ to $11.16$ and three-run mean AR speedup from $6.14{\times}$ to $6.60{\times}$. These show that parent-conditioned branching can turn conditional capacity already present in a semi-autoregressive drafter into end-to-end inference gains through an inference-only change.

Problem

• speculative decoding
• semi-autoregressive
• token acceptance
• inference acceleration
• drafting strategy

Innovation

• speculative decoding
• semi-autoregressive
• tree-based drafting
• parent-conditioned branching
• LLM inference acceleration