Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR

2026-09-29

AI Summary

This study addresses the challenges of credit assignment difficulty and unstable value estimation caused by sparse rewards in multi-step reasoning with large language models (LLMs). To this end, we propose πPPO, a self-privileged Actor-Critic framework. This method reconstructs state value estimation by leveraging successful and failed verification trajectories under the same prompt as contrastive evidence, and introduces an asymmetric critic architecture to facilitate the evaluation of intermediate step quality. Experimental results demonstrate that πPPO significantly improves the accuracy and stability of value estimation, outperforming mainstream baselines on mathematical reasoning benchmarks. Furthermore, the framework remains compatible with small-scale critics, offering an efficient new paradigm for optimizing LLM reasoning via reinforcement learning.

Abstract

Assigning credit to intermediate steps remains a central challenge in training Large Language Models (LLMs) on multi-step reasoning tasks with sparse terminal rewards, and actor-critic methods such as PPO address this by learning value functions to construct token-level advantages. Their effectiveness, however, hinges on reliable value estimation, a difficult task requiring the critic to both assess progress toward a correct solution and anticipate an evolving policy's future behavior; errors in either can compromise credit assignment and destabilize online training. In this paper, we revisit the standard state-only formulation of value estimation and propose $π$PPO, a self-privileged actor-critic framework. By reusing verified same-prompt rollouts as contrastive evidence, $π$PPO helps the critic assess intermediate reasoning against successful and failed attempts, while preserving standard policy optimization and the deployment interface. Experiments show that $π$PPO consistently improves value-estimation quality by a substantial margin and outperforms representative actor-critic and critic-free RLVR baselines on challenging mathematical reasoning benchmarks, while remaining effective even when paired with substantially smaller asymmetric critics.

Problem

• Value Estimation
• Credit Assignment
• Reinforcement Learning
• Large Language Models
• Multi-step Reasoning

Innovation

• Self-Privileged Critic
• Value Estimation
• Actor-Critic
• RLVR
• Contrastive Evidence