From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning

2026-05-31

AI Summary

This work addresses the challenges of explicit reward engineering and low sample efficiency in offline preference-based reinforcement learning by proposing a novel framework that first learns latent successor measure representations from reward-free offline data and then fine-tunes these representations using contrastive search guided by preference feedback. By integrating reward-free representation learning with preference-based policy optimization, the method substantially improves sample efficiency—the first approach to bridge these two paradigms. Empirical evaluations across multiple benchmark tasks demonstrate that the proposed framework consistently outperforms existing offline preference-based reinforcement learning methods, highlighting its superior ability to leverage limited preference signals effectively.

Abstract

Preference-based reinforcement learning (PbRL) avoids explicit reward engineering by learning from pairwise human preference feedback. Existing offline PbRL methods typically follow a two-stage pipeline, first learning a reward or preference model from labeled preferences and then performing offline RL on unlabeled data. We revisit offline PbRL through the lens of reward-free representation learning (RFRL) from the zero-shot RL literature, and propose a new training framework that first learns latent successor-measure representations from reward-free offline data, followed by contrastive search and fine-tuning using preference data. Through extensive experiments and ablations, we show that our method achieves superior preference efficiency over offline PbRL baselines. This work is the first to connect RFRL with PbRL, highlighting its potential as a feedback-efficient solution. Our code is publicly available at https://github.com/rl-bandits-lab/FB-PbRL.

Problem

• Preference-based Reinforcement Learning
• Offline Reinforcement Learning
• Reward-Free Representation Learning
• Human Feedback
• Preference Efficiency

Innovation

• Reward-Free Representation Learning
• Preference-Based Reinforcement Learning
• Successor Measures
• Offline RL
• Contrastive Search