🤖 AI Summary
This work addresses the sharp decline in acceptance rates of speculative decoding when increasing the speculation budget under factorized draft models, a limitation stemming from their independence assumptions that undermines decoding efficiency. To overcome this, the authors propose Weaver—a lightweight autoregressive adapter that constructs a proposal tree from the top-K marginal distributions of the factorized model, thereby restoring inter-token conditional dependencies without requiring full-vocabulary projections. Weaver introduces a novel paradigm for marginal-distribution-based proposal trees and incorporates a rollback-free tree verification algorithm, ensuring high generation quality while accelerating inference. Implemented with Gated Delta Net and optimized CUDA kernels via SGLang, Weaver achieves a 4.37× speedup over standard autoregressive decoding and improves throughput by 24.7% compared to the DFlash baseline.
📝 Abstract
Speculative decoding greatly increases the interactivity of autoregressive language models by trading off computation for extra tokens generated in a single forward pass. Factorized draft models are especially efficient because they predict future-token marginals in parallel, but their independence assumption causes acceptance rates to degrade sharply as the speculative budget grows. We analyze this limitation and introduce Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top-K marginals of a factorized drafter. Weaver restores conditional dependencies between proposed tokens while avoiding a full-vocabulary projection. To support fast verification for models with Gated Delta Net layers, we derive a rollback-free tree-verification algorithm and implement optimized CUDA kernels in SGLang. By combining these model and systems contributions we achieve a 4.37-fold speedup over autoregressive decoding, and outperform a highly optimized DFlash baseline by 24.7%.