Reward Learning from Multiple Feedback Types

2025-02-28

AI Summary

This paper addresses the overreliance on single-type preference feedback in Reinforcement Learning from Human Feedback (RLHF), proposing the first systematic framework for jointly modeling six heterogeneous human feedback types—including ratings, rankings, corrections, and others. Methodologically, it introduces a multi-type-adaptive reward modeling architecture and an end-to-end RL fine-tuning pipeline, trained controllably using high-quality synthetic feedback. Its core contribution is the first empirical and theoretical demonstration that diverse feedback modalities yield synergistic gains in information complementarity, bias mitigation, and generalization—establishing the foundational principles for multi-source feedback RLHF. Experiments across ten RL benchmarks show that the proposed approach improves reward model accuracy by an average of 12.7% over preference-only baselines, with corresponding significant gains in policy performance.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Learning rewards from preference feedback has become an important tool in the alignment of agentic models. Preference-based feedback, often implemented as a binary comparison between multiple completions, is an established method to acquire large-scale human feedback. However, human feedback in other contexts is often much more diverse. Such diverse feedback can better support the goals of a human annotator, and the simultaneous use of multiple sources might be mutually informative for the learning process or carry type-dependent biases for the reward learning process. Despite these potential benefits, learning from different feedback types has yet to be explored extensively. In this paper, we bridge this gap by enabling experimentation and evaluating multi-type feedback in a broad set of environments. We present a process to generate high-quality simulated feedback of six different types. Then, we implement reward models and downstream RL training for all six feedback types. Based on the simulated feedback, we investigate the use of types of feedback across ten RL environments and compare them to pure preference-based baselines. We show empirically that diverse types of feedback can be utilized and lead to strong reward modeling performance. This work is the first strong indicator of the potential of multi-type feedback for RLHF.

Problem

• Learning rewards from diverse human feedback types
• Exploring multi-type feedback for reward modeling
• Evaluating diverse feedback in reinforcement learning environments

Innovation

• Simulates six diverse feedback types for learning.
• Implements reward models for multiple feedback types.
• Compares multi-type feedback across ten RL environments.