Hierarchical Latent Prediction for Language Models

2026-08-06

AI Summary

Standard next-token prediction is limited in long-horizon reasoning by its narrow receptive field and the accumulation of multi-step errors. This work proposes a hierarchical latent variable prediction mechanism that introduces high-level abstract latent variables for the first time, integrating self-supervised latent space prediction, multi-step rollout, and hierarchical representation learning. This approach effectively mitigates error propagation during latent space rollouts and enhances the model’s capacity to maintain coherent, long-range belief states. Evaluated on code generation and multi-step reasoning benchmarks, the method significantly outperforms existing approaches and substantially improves speculative decoding efficiency.

Abstract

While standard Next-Token Prediction (NTP) lays the foundation of language model pre- training, its teacher-forced training paradigm may not be optimal for long-horizon reasoning and planning. Recent works such as Multi-Token Prediction (MTP) and Next-Latent prediction (NextLat) try to mitigate the problem through predicting multiple future tokens and self-supervised prediction in the latent space. However, those auxiliary objectives either have a limited horizon or suffer from compounding error from multi-step rollout. We introduce Hierarchical Latent Prediction (HiLP), which introduces an auxiliary higher-level abstract latent to help reduce the error accumulation effect in latent-space rollouts. Experiments show that HiLP can lead to longer-horizon coherent belief state representation and demonstrate the effectiveness of our method across coding and multi-step reasoning benchmarks, and offers more speculative decoding efficiency.

Problem

• Next-Token Prediction
• long-horizon reasoning
• latent-space rollout
• error accumulation
• language model pre-training

Innovation

• Hierarchical Latent Prediction
• latent-space rollout
• error accumulation
• long-horizon reasoning
• speculative decoding