SLPO: Scaling Latent Reasoning via a Surrogate Policy

2026-07-21

AI Summary

Existing implicit reasoning methods struggle to leverage outcome-based rewards for reinforcement learning due to the absence of computable stepwise likelihoods and adaptive stopping mechanisms, limiting their test-time scalability. This work proposes Surrogate Latent Policy Optimization (SLPO), the first approach to integrate outcome-reward-based reinforcement learning into autoregressive implicit reasoners. SLPO constructs an empirical surrogate policy density via latent variable modeling to enable trajectory-level credit assignment and introduces a correctness-supervised stopping head for dynamic control of reasoning length. By overcoming the reliance on imitation learning inherent in prior implicit reasoning frameworks, SLPO significantly improves Pass@$k$ performance under both continuous and soft reasoning settings and adaptively allocates more reasoning time to challenging instances, thereby enhancing deterministic accuracy.

Abstract

Reinforcement learning with verifiable rewards has become the predominant recipe for eliciting test-time scaling in explicit Chain-of-Thought reasoners. Yet this scaling path remains computationally costly, since every intermediate step must be decoded as a language token. Latent reasoning instead carries intermediate computation as continuous vectors and already matches or surpasses explicit CoT at far shorter horizons. Despite this promise, latent reasoners remain largely imitation-bound, while explicit CoT has already moved past imitation via outcome-reward RL. Latent trajectories lack a tractable per-step likelihood and an adaptive stopping interface under fixed thinking budgets, so outcome rewards cannot elicit latent test-time scaling. We introduce Surrogate Latent Policy Optimization (SLPO) to bring outcome-reward RL to autoregressive latent reasoners: an empirical surrogate policy density over latent transitions for trajectory-level credit assignment, and a correctness-supervised stopping head that outcome-reward optimization refines into a variable-horizon policy. Across continuous and soft thinking settings, SLPO improves Pass@$k$ under parallel sampling and allocates longer latent computation to harder instances with higher deterministic accuracy.

Problem

• latent reasoning
• outcome-reward RL
• test-time scaling
• autoregressive latent reasoners
• adaptive stopping

Innovation

• Latent Reasoning
• Reinforcement Learning
• Surrogate Policy
• Test-time Scaling
• Autoregressive Reasoning