ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning

2026-07-29

AI Summary

This work addresses the challenge of accurate token-level credit assignment in mathematical reasoning tasks, where standard Proximal Policy Optimization (PPO) suffers from sparse rewards and long reasoning chains, leading to biased value estimates and suboptimal policies. To overcome this, the authors propose the ReDiPPO framework, which introduces a reference-answer-guided critic to calibrate value estimation. By computing token-level discrepancies between the standard and reference critics, ReDiPPO identifies difficult reasoning states and dynamically reweights the advantage function to adaptively emphasize critical reasoning steps. Evaluated on multiple mathematical reasoning benchmarks, ReDiPPO significantly outperforms strong baselines—including PPO, DAPO, and GSPO—demonstrating improved value estimation accuracy and enhanced final reasoning performance.

Abstract

Reinforcement learning has emerged as an effective paradigm for enhancing the mathematical reasoning capabilities of large language models. Among existing policy optimization methods, Proximal Policy Optimization (PPO) remains particularly appealing because its learned critic can, in principle, provide token-level credit assignment. However, in mathematical reasoning tasks characterized by long reasoning horizons and sparse outcome rewards, reliable token-level credit assignment remains challenging. The standard critic often fails to accurately evaluate intermediate reasoning states, resulting in noisy advantage estimates and suboptimal policy updates. In this paper, we propose ReDiPPO, a Reference-guided and Discrepancy-aware PPO framework for mathematical reasoning. ReDiPPO introduces a reference-guided critic that uses reference answers as training-time privileged signals to provide more accurate value estimation. Meanwhile, it retains a standard critic and quantifies the token-level reference-standard discrepancy between the standard value estimate and the reference-guided value estimate. This discrepancy serves as an indicator of difficult reasoning states and is used to reweight the corresponding token-level advantages during PPO optimization. Extensive experiments on diverse mathematical reasoning benchmarks demonstrate that ReDiPPO improves value-estimation accuracy and consistently outperforms strong policy optimization baselines, including PPO, DAPO, and GSPO, in final reasoning performance. Our code is available on https://github.com/cii030/ReDiPPO.

Problem

• mathematical reasoning
• credit assignment
• value estimation
• reinforcement learning
• sparse rewards

Innovation

• Reference-Guided Critic
• Discrepancy-Aware Reweighting
• Token-Level Credit Assignment
• Mathematical Reasoning
• Proximal Policy Optimization