Latent Thought Credit: Multi-Answer Credit Assignment for Latent Reasoning

2026-08-02

AI Summary

This work addresses the challenge of credit assignment in latent reasoning processes of language models when only final-answer rewards are available. The authors propose the Latent Thought Credit (LTC) framework, which estimates step-level expected rewards by sampling multiple reasoning paths per prompt and generating multiple answers under fixed intermediate contexts to average the observed rewards. LTC mitigates credit assignment errors through this multi-answer reward estimation, enhances reproducibility of high-value reasoning paths via an advantage-weighted reasoning matching objective, and integrates hierarchical credit assignment with fixed-context diagnostics using a GRPO-style on-policy training scheme. Experiments demonstrate that LTC achieves state-of-the-art average accuracy on mathematical and STEM multiple-choice tasks, and ablation studies confirm its effectiveness in alleviating ambiguity in reasoning credit assignment.

Abstract

Latent reasoning allows language models to carry out intermediate reasoning in continuous latent representations rather than fully externalizing it as discrete chains of thought. However, assigning credit to such latent thoughts from answer-only rewards is difficult: a single final answer mixes thought quality with answer-sampling noise. We propose \textbf{Latent Thought Credit (LTC)}, a hierarchical credit-assignment framework for latent reasoning. For each prompt, LTC samples multiple latent thoughts, fixes the context after each thought, and estimates thought-level expected reward by averaging rewards over multiple answers generated from that fixed context. LTC uses thought-level advantages to optimize the latent-thought phase, answer-level advantages to optimize the answer phase, and an advantage-weighted thought-matching objective that helps the policy reproduce high-credit latent thoughts. We instantiate LTC in a GRPO-style on-policy training framework and evaluate it across mathematical reasoning and STEM multiple-choice tasks. LTC achieves the best average accuracy among the compared methods, while ablations and fixed-context diagnostics show that multi-answer estimation reduces reward-estimation error and mitigates ambiguous or incorrect thought-level credit.

Problem

• latent reasoning
• credit assignment
• reward estimation
• thought quality
• answer-only rewards

Innovation

• Latent Reasoning
• Credit Assignment
• Multi-Answer Estimation
• Hierarchical Reinforcement Learning
• Thought-Level Advantage