The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement

2026-05-29

AI Summary

This work addresses the limitations of existing reward models, which suffer from the high cost and limited diversity of human preference data and struggle to generalize to new responses generated by evolving policies. To overcome these challenges, the authors propose SAVE, a framework that leverages policy value functions to score online-generated responses and construct self-supervised signals for continuously refining the reward model. Key innovations include a prompt-specific online feedback mechanism grounded in value anchoring, ambiguous sample filtering, a contrastive learning objective, and a reward model advantage computation. Experimental results demonstrate that SAVE consistently outperforms baseline methods across six diverse benchmarks and yields robust performance gains when integrated with three distinct reinforcement learning algorithms—GRPO, RLOO, and GSPO—under varying policy architectures.

Abstract

Building strong reward models (RMs) for language model alignment is bottlenecked by the cost and difficulty of acquiring diverse and reliable preference data from human annotation or judge models. It is dramatically worse as the policy evolves beyond the static RM training. Therefore, we propose SAVE (Self-supervised reward model improvement via Value-Anchored On-policy feedback), a framework that grades on-policy responses as feedback by using the value function for on-policy RM training. SAVE naturally converts the reward-graded on-policy responses into supervision with a prompt-specific value head as an adaptive anchor. It computes RM advantages and filters ambiguous samples to update the RM via a contrastive objective. The effectiveness of SAVE for enhancing RM training is strongly validated through rigorous empirical evaluation across six diverse benchmarks. It achieves outperforming results across all datasets while maintaining consistent improvements across three RL algorithms (GRPO, RLOO, GSPO) and different policy backbones.

Problem

• reward modeling
• preference data
• language model alignment
• on-policy feedback
• human annotation

Innovation

• reward modeling
• self-supervised learning
• on-policy feedback
• value anchoring
• contrastive learning