AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

2026-07-28

AI Summary

Existing single-draft models struggle to accommodate the highly divergent output distributions across diverse large language model inference tasks, limiting speculative decoding efficiency. This work proposes AngelSpec, a unified training framework that jointly optimizes multi-token prediction (MTP) and block-wise parallel diffusion draft models through co-specialization of architecture and data—a first in the field. It introduces the DFly architecture, which integrates a hybrid-conditioned backbone with a look-ahead conditional autoregressive head to enable efficient parallel generation. Furthermore, verification is treated as a batch-level shared resource, with verification depth dynamically scheduled via a utility-cost model. Experiments show that on Hy3-A21B, AngelSpec achieves an average accepted length improvement of approximately 30%, consistently attains peak throughput across concurrency levels from 4 to 64, and accelerates decoding by 1.98–2.40× over autoregressive baselines, outperforming DFlash by 10.5–11.8%.

Abstract

Speculative decoding accelerates large language model inference without changing the target distribution, but no single drafting structure performs best across real-world workloads. Autoregressive multi-token prediction (MTP) is a lightweight, stable proposal mechanism, whereas block-parallel diffusion amortizes drafting latency over much longer candidate sequences; the better choice depends strongly on the output distribution. We present AngelSpec, a unified training framework for MTP and block-parallel speculative decoding that addresses this heterogeneity at three levels. At the training level, rather than fitting one universal drafter to a uniform data mixture, we co-specialize structure and data: the MTP drafter is trained on diverse conversational data for high-entropy open-ended chat, and the block-diffusion drafter on code and mathematics data for longer predictable continuations. At the architecture level, we propose DFly, a block-diffusion framework combining a hybrid target-conditioning backbone with a predecessor-conditioned autoregressive head, improving target-feature utilization and intra-block dependency modeling while keeping generation parallel. At the inference level, both acceptance length and verification cost vary with domain, request, online load, and hardware, so DFly treats verification as a shared batch-level resource: it reallocates compute toward high-confidence prefixes across requests and combines expected utility with a profiled cost model to adapt verification depth online. Across the Hy3 series, DFly raises the average accepted length on Hy3-A21B by roughly 30% and attains the highest average throughput at every tested concurrency from 4 to 64, a 1.98-2.40x speedup over autoregressive decoding and 10.5-11.8% higher throughput than DFlash. We release AngelSpec to support training and extending these methods.

Problem

• speculative decoding
• large language model inference
• multi-token prediction
• block-parallel diffusion
• real-world workloads

Innovation

• speculative decoding
• multi-token prediction
• block-parallel diffusion
• adaptive verification
• unified training framework