Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problems

2026-05-29

AI Summary

This work addresses the inefficiency and lack of theoretical guarantees in preference-based reinforcement learning for long-horizon decision-making. To this end, we propose the Markov Decision Competition (MDC) framework, which formalizes the learning objective as pairwise preferences over policies rather than scalar rewards. Our theoretical analysis establishes that, under this model, static Markov policies are optimal among all history-dependent policies, and exact solution computation lies in complexity class P. Building on these insights, we develop an iterative optimization algorithm and prove its convergence. Empirical results demonstrate that our approximate algorithm substantially outperforms existing approaches on high-dimensional, long-horizon tasks, achieving significant improvements in both sample efficiency and policy performance.

Abstract

Reinforcement learning problems typically define the goal as maximizing the expected value of a scalar reward function. But, pairwise preferences are often easier to specify than scalar rewards, and they express certain goals that scalar rewards cannot. Methods for reinforcement learning with pairwise preferences have thus received growing interest. Unfortunately, these methods are inefficient in problems with long time horizons, and they lack guarantees on the performance of Markov policies relative to history-dependent policies, which bridge the theory and practice of reinforcement learning. We therefore propose the \textit{Markov decision contest} as a new problem model for reinforcement learning with pairwise preferences. We prove that stationary Markov policies are optimal among all history-dependent policies, that solving a Markov decision contest exactly is in P, and that a simple iterative algorithm converges to an optimal policy at a sublinear rate. Lastly, in a set of high-dimensional decision problems with long time horizons, we show that our approximate algorithm is significantly more learning-efficient than prior work.

Problem

• reinforcement learning
• pairwise preferences
• long-term decision problems
• Markov policies
• history-dependent policies

Innovation

• Markov decision contest
• pairwise preferences
• reinforcement learning
• stationary Markov policies
• learning efficiency