Relative Value Learning

2026-07-23

AI Summary

This work proposes a relative value learning framework that shifts the focus from absolute value estimation—commonly used in traditional reinforcement learning—to directly modeling pairwise relative value differences, which are sufficient for policy optimization. The framework introduces an antisymmetric function to represent value differences between state pairs and defines a novel pairwise Bellman operator with a unique fixed point. Building upon this foundation, the authors derive n-step and λ-return objectives and develop an unbiased Relative Generalized Advantage Estimator (R-GAE) for policy gradient computation. When integrated into Proximal Policy Optimization (PPO), the approach achieves performance on par with standard PPO across 49 Atari games, demonstrating that relative value estimation can serve as an effective alternative to absolute value critics.

Abstract

In reinforcement learning, critics typically estimate absolute state values $V(s)$, estimating how good a particular situation is in isolation. However, it turns out that only differences in value are relevant for control. Motivated by this, we propose Relative Value Learning (RV), a framework that learns value differences directly via an antisymmetric function $Δ(s_i, s_j) = V(s_i) - V(s_j)$. We introduce a pairwise Bellman operator and prove it is a $γ$-contraction with a unique fixed point equal to the true value differences, derive well-posed $1$-step, $n$-step and $λ$-return targets and reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE). Beyond theoretical results, we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 ALE games) compared to standard PPO, indicating that relative value estimation is an effective alternative to absolute critics.

Problem

• Relative Value Learning
• Reinforcement Learning
• Value Differences
• Critic Estimation
• Policy Gradient

Innovation

• Relative Value Learning
• Pairwise Bellman Operator
• Antisymmetric Value Function
• Generalized Advantage Estimation
• Policy Gradient