RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction

2026-08-06

AI Summary

This work addresses the challenge that generative reward models, due to their comparative outputs, are incompatible with the scalar rewards required by reinforcement learning, thereby hindering effective training of large language models. To overcome this limitation, the paper proposes a Ranking-based Reward Construction (RRC) method, which innovatively introduces self-competitive ranking and anchor-guided ranking strategies to transform relative preference rankings into scalar reward signals suitable for reinforcement learning. By circumventing the constraints of conventional scalar reward formulation, RRC achieves substantially improved training performance on open-ended dialogue and reasoning benchmarks, consistently outperforming existing reward modeling approaches.

Abstract

Recent advances in reward modeling show a paradigm shift from discriminative reward models to generative reward models. However, despite their strong capabilities in response ranking, generative reward models have not realized their potential in reinforcement learning (RL). Our analysis reveals that this limitation arises from a mismatch between the comparative nature of generative reward modeling and the scalar scoring paradigm adopted by existing RL algorithms. To bridge this gap, we propose a Ranking-based Reward Construction (RRC) approach, which enables generative reward models to provide more effective RL learning signals by deriving rewards from relative preference rankings. RRC introduces two complementary strategies: self-competitive ranking, which exploits comparisons among sampled responses, and anchor-guided ranking, which enables scalable ranking-based reward construction with a small set of reference responses. Experiments across open-ended chat and reasoning benchmarks demonstrate that RRC substantially improves RL training with generative reward models, achieving consistent gains over existing reward construction approaches. Our code can be found at https://github.com/wangclnlp/RRC.

Problem

• generative reward models
• reinforcement learning
• reward modeling
• ranking
• scalar scoring

Innovation

• generative reward models
• ranking-based reward construction
• reinforcement learning
• preference ranking
• LLM alignment