autoregressive decoding

Autoregressive (AR) text generation in large language models (LLMs) suffers from slow inference due to sequential token prediction. Method: This paper systematically surveys and restructures the parallel text generation landscape, proposing the first unified taxonomy encompassing AR parallel decoding, non-autoregressive (Non-AR) modeling, diffusion-based language models, and knowledge distillation. Through theoretical analysis and empirical benchmarking across standard datasets and real-world scenarios, it characterizes the speed–quality–efficiency trade-offs inherent in each paradigm. Contribution/Results: The study identifies key acceleration pathways and synergistic integration opportunities, establishes performance boundaries, summarizes state-of-the-art advances, and highlights persistent challenges—including scalability, output consistency, and generalization. It delivers the first structured technical roadmap for efficient LLM inference, advancing the paradigm shift from serial to parallel generation.

Autoregressive decoding in large language models (LLMs) lacks backtracking capability, leading to suboptimal sequences that deviate from the globally optimal joint probability distribution. To address this, we propose an uncertainty-aware selective refinement framework. Our method introduces token-level uncertainty estimation, integrated with a sliding reflective window and a dynamic pausing mechanism, enabling localized resampling and re-decoding during generation. The framework is plug-and-play—requiring no architectural modifications—and preserves high inference efficiency while approaching joint-probability optimality. Evaluated across diverse open-ended generation and reasoning benchmarks, it significantly improves factual consistency and fluency, yielding BLEU and ROUGE gains of 2.1–4.3 points, with inference latency overhead under 12%.

This work addresses the challenge of applying autoregressive generative models to low-latency streaming target speaker extraction, a task hindered by their inherent reliance on global context. The study proposes a novel chunk-wise interleaved concatenation paradigm to enable efficient real-time inference and introduces a historical context refinement mechanism to mitigate discontinuities at speech segment boundaries. Evaluated on the Libri2Mix dataset, the system achieves 100% inference stability and high speech intelligibility, matching or even surpassing offline baselines in streaming performance. Notably, it attains a real-time factor (RTF) of 0.248 on consumer-grade GPUs, demonstrating practical feasibility for real-world deployment.

This work addresses the absence of a unified theoretical framework for autoregressive decoding strategies in speech processing, which has led to ambiguous definitions, inconsistent taxonomies, and difficulties in fair comparison. The paper introduces, for the first time, a general formal framework that precisely specifies inclusion criteria for autoregressive search and systematically categorizes and describes decoding strategies employed in neural speech generation models. By clarifying conceptual boundaries, the framework enhances comparability and evaluation consistency across strategies, streamlines the design of decoding-centric benchmarking protocols, and enables ablation studies focused specifically on search mechanisms. Consequently, it facilitates standardized analysis of inference-stage behavior in speech generation models.

Continuous autoregressive visual generation models suffer from high inference latency, while existing speculative decoding methods are restricted to discrete token spaces and lack theoretical foundations or practical techniques for continuous-valued outputs. Method: This work pioneers the extension of speculative decoding to continuous visual generation. We propose a diffusion-prior-based continuous acceptance criterion, design a denoising trajectory alignment mechanism and token pre-filling strategy to mitigate distribution mismatch, and establish a continuous accept-reject sampling framework with analytically derived upper bounds on approximation error. Contribution/Results: Our approach achieves a 2.33× inference speedup on standard diffusion-based autoregressive models while provably preserving the exact output distribution of the original model. The implementation is publicly available.

To address the slow autoregressive decoding of large language models (LLMs) and the low acceptance rates and limited speedup of existing speculative decoding methods—caused by rigid, fixed-structure draft sequences—this paper proposes Dynamic Optimal Draft Tree Speculative Decoding. We formulate the single-step expected accepted token length as the optimization objective and adaptively construct an extensible draft tree structure, thereby overcoming the limitations of static tree designs. Our method integrates a probability-driven tree search algorithm, a lightweight autoregressive draft model, and an efficient verification mechanism to enable parallel multi-token generation with lossless acceleration. Experiments across diverse LLMs and tasks demonstrate up to 3.2× decoding speedup over standard autoregressive decoding, with an average of over 10 tokens accepted per step—significantly outperforming state-of-the-art draft strategies.

This work addresses the sharp decline in acceptance rates of speculative decoding when increasing the speculation budget under factorized draft models, a limitation stemming from their independence assumptions that undermines decoding efficiency. To overcome this, the authors propose Weaver—a lightweight autoregressive adapter that constructs a proposal tree from the top-K marginal distributions of the factorized model, thereby restoring inter-token conditional dependencies without requiring full-vocabulary projections. Weaver introduces a novel paradigm for marginal-distribution-based proposal trees and incorporates a rollback-free tree verification algorithm, ensuring high generation quality while accelerating inference. Implemented with Gated Delta Net and optimized CUDA kernels via SGLang, Weaver achieves a 4.37× speedup over standard autoregressive decoding and improves throughput by 24.7% compared to the DFlash baseline.

Autoregressive decoding in large language models incurs high latency, and existing multi-token prediction methods rely on strong independence assumptions that limit modeling fidelity. Method: We propose Parallel Token Prediction (PTP), the first framework to internalize the sampling process into the model architecture, enabling joint generation of multiple semantically coherent tokens in a single Transformer forward pass while strictly preserving expressivity over any autoregressive distribution—thereby eliminating restrictive independence assumptions. PTP integrates inverse autoregressive training with explicit sampling modeling and supports both teacher-free and distillation-based training. Results: Evaluated on Vicuna-7B, PTP achieves 4.12 average accepted tokens per speculative step on Spec-Bench, maintains full modeling capability for long-sequence generation, and attains state-of-the-art performance in speculative decoding.

Traditional autoregressive language models often suffer from output instability, repetition, and sensitivity to decoding strategies due to premature token discretization at each generation step. This work proposes a continuous autoregressive generation framework that represents tokens as continuous vectors and allows them to undergo multi-step dynamic evolution before committing to a hard decoding decision. For the first time, this approach enables autoregressive text generation without per-token sampling, relying instead solely on deterministic argmax operations—augmented optionally by stochastic dynamics and historical smoothing mechanisms. Once converged, the method produces coherent and diverse text while significantly reducing reliance on explicit sampling or auxiliary stabilization techniques.

Autoregressive image generation suffers from low information density and spatially uneven distribution of image tokens, limiting both generation quality and decoding speed. To address this, we propose an entropy-driven efficient decoding framework. Our key contributions are: (1) a spatial entropy-guided dynamic temperature scheduling mechanism that balances token diversity and structural consistency; (2) an entropy-aware acceptance criterion for speculative decoding, significantly improving token acceptance reliability; and (3) a lightweight design compatible with both mask-based and scale-wise autoregressive architectures. Extensive experiments across multiple benchmarks and model variants demonstrate that our method preserves near-lossless generation quality while reducing inference cost to 85% of conventional acceleration approaches—outperforming existing decoding strategies in both efficiency and fidelity.

This work addresses the inefficiency of autoregressive image generation models, which suffer from slow inference due to sequential token-by-token decoding and error propagation inherent in existing acceleration methods constrained to one-dimensional sequences. The study proposes, for the first time, a training-free parallel decoding strategy that extends Jacobi-style decoding into the two-dimensional spatial domain. By leveraging local spatial correlations in images, the method predicts candidate tokens in parallel within two-dimensional neighborhoods and introduces specially designed attention masks to ensure convergence stability. This approach effectively mitigates error accumulation, achieving 4.8–6.4× inference speedup across multiple autoregressive models and datasets while maintaining competitive generation quality.