Distributional Inverse Reinforcement Learning

2025-10-03

AI Summary

Traditional offline inverse reinforcement learning (IRL) models only deterministic rewards or expected returns, failing to capture the distributional characteristics of rewards and returns—or expert risk preferences—in demonstrated behavior. To address this, we propose the first *distributional* offline IRL framework that jointly models uncertainty in both the reward function and the return distribution. Our method captures structural diversity in expert behavior by minimizing violations of first-order stochastic dominance and—uniquely in IRL—incorporates distorted risk measures to jointly recover reward distributions and risk-aware policies. This breaks the limitations of expectation-centric and deterministic modeling paradigms. Empirically, our approach achieves state-of-the-art imitation performance across synthetic environments, real neural behavioral datasets, and MuJoCo benchmarks, significantly improving reward representation fidelity and policy robustness.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We propose a distributional framework for offline Inverse Reinforcement Learning (IRL) that jointly models uncertainty over reward functions and full distributions of returns. Unlike conventional IRL approaches that recover a deterministic reward estimate or match only expected returns, our method captures richer structure in expert behavior, particularly in learning the reward distribution, by minimizing first-order stochastic dominance (FSD) violations and thus integrating distortion risk measures (DRMs) into policy learning, enabling the recovery of both reward distributions and distribution-aware policies. This formulation is well-suited for behavior analysis and risk-aware imitation learning. Empirical results on synthetic benchmarks, real-world neurobehavioral data, and MuJoCo control tasks demonstrate that our method recovers expressive reward representations and achieves state-of-the-art imitation performance.

Problem

• Models reward uncertainty and full return distributions in offline IRL
• Captures expert behavior structure using stochastic dominance and risk measures
• Enables risk-aware imitation learning and expressive reward recovery

Innovation

• Distributional framework models reward and return uncertainties
• Minimizes FSD violations using distortion risk measures
• Recovers expressive reward distributions and risk-aware policies