K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modeling

2026-06-09

AI Summary

Autoregressive language models suffer from inefficient inference due to token-by-token decoding, limiting their applicability in high-throughput batch processing scenarios. This work proposes K-Forcing, a forward-pushing language modeling paradigm that leverages progressive self-forced distillation to transform an autoregressive teacher model into a conditional mapping capable of generating k future tokens in parallel within a single forward pass. Built upon the causal Transformer architecture, the method maps uniform noise to joint multi-token samples while remaining compatible with existing autoregressive deployment pipelines. Experiments on LM1B and OpenWebText demonstrate that with k=4, K-Forcing achieves 2.4–3.5× inference speedup over standard autoregressive decoding, accompanied by only minor degradation in generation quality, thereby substantially enhancing inference efficiency in high-throughput settings.

Abstract

Autoregressive (AR) language modeling is the dominant paradigm for text generation, yet its sequential token-by-token decoding makes inference memory-bound and inefficient. Existing acceleration approaches, such as speculative decoding and diffusion language models, can yield speedups under certain conditions but do not directly address high-load batch serving--the scenario most critical for industrial-scale deployment. We introduce K-Forcing, a push-forward language modeling paradigm for joint next-k-token decoding. K-Forcing distills an existing AR model into a conditional push-forward mapping--one that transforms independent uniform noise variables into a joint sample of multiple future tokens in a single forward pass. This design preserves fixed-length outputs, reuses the AR teacher backbone, and remains compatible with standard AR serving infrastructure. We train this mapping via progressive self-forcing distillation, which gradually expands the prediction window while enabling the student to closely match the sequence distribution of the AR teacher. We evaluate K-Forcing on LM1B and OpenWebText using a standard causal Transformer backbone. When aggressively configured to generate k = 4 tokens per forward pass, K-Forcing delivers approximately 2.4-3.5x speedup across different batch sizes, while incurring modest quality degradation relative to its AR teacher. As inference increasingly dominates the lifetime compute cost of modern LLMs, K-Forcing offers a promising route toward accelerating AR generation under real-world high-load deployment.

Problem

• autoregressive language modeling
• text generation
• inference efficiency
• high-load batch serving
• token-by-token decoding

Innovation

• K-Forcing
• push-forward language modeling
• joint next-k-token decoding
• progressive self-forcing distillation
• autoregressive acceleration