WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing

2026-06-05

AI Summary

This work addresses the inefficiency of autoregressive inference in large language models, which is exacerbated by the static draft mechanisms in existing speculative decoding approaches that fail to adapt to sharp intra-sequence variations in draft accuracy. To overcome this limitation, we propose WhiFlash, the first framework to dynamically fuse autoregressive and diffusion-style parallel drafting at the token level. WhiFlash employs a fine-grained routing strategy driven jointly by entropy-aware signals and a lightweight neural controller to select the optimal drafting paradigm in real time. Additionally, it incorporates Lazy Catch-up and KV-only Prefill caching optimizations to limit paradigm-switching overhead to under 7% of per-step latency. Experiments demonstrate that WhiFlash achieves throughput improvements of up to 69.6% over EAGLE-3 and 37.3% over DFlash.

Abstract

The autoregressive nature of large language models (LLMs) remains a significant bottleneck for inference, particularly in complex agentic workloads. While speculative decoding (SD) accelerates inference, current approaches rely on static drafting paradigms, utilising either autoregressive drafting models for reasoning or diffusion-based parallel drafting models for structured outputs. We empirically find that drafting accuracy fluctuates dramatically within a single sequence, leaving significant performance unrealised by static paradigms and coarse-grained routing. To address this volatility, we introduce WhiFlash, the first cross-paradigm SD method that unifies autoregressive and diffusion-based parallel drafting under a single token-level controller. WhiFlash adopts a fine-grained routing mechanism that employs either a lightweight entropy-based or a learned neural policy, both parametrised to provide a tunable balance between expected token gain and latency. To make high-frequency switching computationally viable, we introduce novel cache-management optimisations, Lazy Catch-up and KV-only Prefill, reducing switching overhead to below 7% of per-round latency. By capitalising on the complementary strengths of fundamentally distinct drafting architectures, WhiFlash achieves significantly higher acceptance lengths, yielding category-specific throughput gains of up to 69.6% over the state-of-the-art autoregressive EAGLE-3 and 37.3% over the diffusion-based DFlash.

Problem

• speculative decoding
• autoregressive models
• diffusion-based drafting
• token-level routing
• inference acceleration

Innovation

• speculative decoding
• cross-paradigm routing
• token-level control
• diffusion-based drafting
• cache optimization