LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback

2026-07-31

AI Summary

Real-world decision-making often involves multiple conflicting objectives, and the true reward function is typically difficult to specify a priori. To address this challenge, this work proposes the LEMUR framework, which extends preference-based reinforcement learning to the multi-objective setting for the first time. LEMUR leverages preference feedback from multiple human evaluators to jointly learn, in an end-to-end manner, both a multi-objective policy and its corresponding reward model, without requiring any pre-defined reward functions. By integrating multi-objective reward modeling with policy optimization, the method achieves a significant performance advantage over existing baselines across several benchmark tasks, effectively balancing and optimizing among competing objectives in an unsupervised setting.

Abstract

Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function. However, real-world decision-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground-truth reward functions are difficult to specify or inaccessible. While Multi-Objective RL (MORL) addresses such trade-offs by modeling rewards as vectors, existing approaches typically assume access to a well-specified reward function for each objective, inheriting the same challenges faced by single-objective RL. Meanwhile, Preference-based RL (PbRL) has shown great potential in solving complex tasks without access to a pre-defined reward function through reward learning from human feedback, yet has largely been studied in single-objective settings. In this work, we bridge this gap with LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback, a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies. Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling agents to effectively balance competing objectives during learning. We evaluate LEMUR on a variety of benchmark multi-objective tasks, and empirical results demonstrate its superior performance over baseline methods. Our method presents a promising direction for solving multi-objective decision-making tasks without pre-defined reward functions.

Problem

• Multi-Objective Reinforcement Learning
• Preference-based RL
• Reward Learning
• Human Feedback
• Decision-making

Innovation

• Multi-Objective Reinforcement Learning
• Preference-based Reinforcement Learning
• Reward Learning from Human Feedback
• Interactive Alignment
• LEMUR