On the Partial Identifiability in Reward Learning: Choosing the Best Reward

2025-01-10

AI Summary

This paper addresses the optimal reward selection problem in reward learning when the target reward function is only partially identifiable. To overcome the limitation of conventional methods—which restrict reward selection to a feasible set and thereby hinder downstream policy generalization—we propose a novel “out-of-feasible-set reward selection” paradigm. We first establish a quantitative identifiability modeling framework grounded in convex analysis and optimization theory, and rigorously prove that there exist rewards outside the feasible set that strictly dominate all feasible ones. Building on this insight, we design three provably efficient algorithms tailored for reward transfer, each accompanied by theoretical guarantees on convergence and sample efficiency. Experiments across multi-task reward transfer benchmarks demonstrate that our approach significantly improves policy generalization performance compared to state-of-the-art baselines.

Technology Category

• Search and Optimization: Learning to Search
• Game Theory and Economic Paradigms: Adversarial Learning
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

In Reward Learning (ReL), we are given feedback on an unknown *target reward*, and the goal is to use this information to find it. When the feedback is not informative enough, the target reward is only *partially identifiable*, i.e., there exists a set of rewards (the feasible set) that are equally-compatible with the feedback. In this paper, we show that there exists a choice of reward, non-necessarily contained in the feasible set that, *depending on the ReL application*, improves the performance w.r.t. selecting the reward arbitrarily among the feasible ones. To this aim, we introduce a new *quantitative framework* to analyze ReL problems in a simple yet expressive way. We exemplify the framework in a *reward transfer* use case, for which we devise three provably-efficient ReL algorithms.

Problem

• Reward Learning
• Partial Observability
• Optimal Reward Selection

Innovation

• Partially Observable Reward Learning
• Decision-making under Uncertainty
• Effective ReL Algorithms