CURE: Local Uncertainty Repair for Block-Parallel Speculative Decoding

2026-08-01

AI Summary

This work addresses the challenge in block-wise parallel speculative decoding where localized high uncertainty in long-sequence generation leads to clustered errors, resulting in elevated rejection rates and diminished speedup. To mitigate this, the authors propose a budget-aware dynamic repair tree structure that identifies uncertainty hotspots at prediction confidence boundaries and constructs bounded repair paths only at critical nodes. A lightweight state resynchronization mechanism is further introduced to enhance acceptance length and verification efficiency with minimal overhead. The method seamlessly integrates with mainstream parallel draft generation frameworks and demonstrates consistent improvements across multiple code and mathematical reasoning benchmarks, achieving a 4.2–7.5% increase in average acceptance length and end-to-end inference speeds of 2.66–3.49× over baseline autoregressive decoding.

Abstract

Speculative decoding mitigates the latency of sequential generation in autoregressive Large Language Models (LLMs) by interleaving draft generation with target verification. However, existing parallel drafting backends often suffer from rapid accuracy degradation over long horizons, leading to high rejection rates during verification and suboptimal wall-clock speedups. We observe that drafting errors are not uniformly distributed but typically stem from localized high-uncertainty tokens that destabilize downstream generation trajectories. Motivated by this token error pattern, we propose CURE, a budget-aware dynamic repair tree designed to repair errors at uncertainty focal points without incurring prohibitive tree-verification overheads. Specifically, our method uses predictive confidence margins to dynamically locate candidate error tokens within a block-parallel draft, expands bounded repair paths only at these fragile nodes, and employs a novel repair resynchronization mechanism to realign draft states post-verification. Evaluations on code-generation benchmarks (HumanEval, MBPP, and LiveCodeBench-lite) and mathematical reasoning benchmark (GSM8K) demonstrate that CURE increases the average accepted length by 4.2-7.5% over parallel baselines without repair, translating to an end-to-end speedup of $2.66-3.49\times$ over target-only decoding. Furthermore, we provide a plug-and-play repair module compatible with standard parallel drafting frameworks. We also characterize the trade-off between draft compute and verification efficiency.

Problem

• speculative decoding
• uncertainty
• parallel drafting
• error propagation
• verification rejection

Innovation

• speculative decoding
• uncertainty-aware repair
• block-parallel generation
• dynamic repair tree
• token-level error correction