SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling

2026-08-03

AI Summary

This work proposes SP3O, a gradient-based preference-based reinforcement learning (PbRL) algorithm that operates without a reward model or critic and, for the first time, supports segment-level preference feedback in general stochastic MDPs. Existing PbRL approaches typically rely on reward models, are limited to simple environments, or require costly trajectory-level feedback. In contrast, SP3O constructs a policy value difference estimator via off-policy importance sampling and updates the policy using a PPO-style objective. Empirical results demonstrate that SP3O substantially reduces human annotation burden while outperforming current PbRL and RLHF methods on long-horizon tasks, including robotic control and large language model fine-tuning.

Abstract

Preference-based reinforcement learning (PbRL) for general stochastic MDPs often requires training a reward model. Existing reward-model-free methods are either restricted to bandits or deterministic MDPs, such as DPO or P3O, or use zeroth-order, gradient-free optimization, which in general exhibits a slower convergence rate than gradient-based algorithms. Furthermore, existing reward-model-free preference-based RL algorithms almost exclusively use trajectory-level feedback, which can require significant effort from a human evaluator when trajectories are long. On the other hand, segments are much shorter, so they are easier to compare and evaluate. In this paper, we introduce a novel reward-model-free, critic-free, and gradient-based PbRL algorithm compatible with segment preferences named Segment Pairwise Proximal Policy Optimization (SP3O). SP3O utilizes segment-level preference feedback to construct an accurate policy value difference estimator via off-policy importance sampling, and then uses the estimator to compute the policy gradient via a PPO-type loss function. We provide a theoretical basis for the algorithm and analyze the tradeoff in choosing the segment length. We also evaluate it experimentally against other PbRL/RLHF algorithms in robotic control and LLM finetuning settings to show its improved performance, especially in long-horizon tasks.

Problem

• preference-based reinforcement learning
• reward-model-free
• segment preferences
• trajectory-level feedback
• stochastic MDPs

Innovation

• reward-model-free
• segment-level preferences
• gradient-based reinforcement learning
• importance sampling
• preference-based RL