Inverse Reinforcement Learning using Revealed Preferences and Passive Stochastic Optimization

2025-07-06

AI Summary

This paper addresses three core challenges in inverse reinforcement learning (IRL): (1) reconstructing utility functions from observed behavior; (2) distinguishing whether an agent is a rational, Bayesian utility maximizer—or instead exhibits rational inattention; and (3) online tracking of time-varying utility functions. We propose a unified framework integrating Afriat’s theorem from revealed preference theory with stochastic optimization via an adaptive Langevin dynamics algorithm, augmented by Bayesian inference and inverse stopping-time modeling to ensure robust identification under noise. Our key contributions include the first IRL method capable of identifying cognitive radar and Bayesian sequential detectors, and the novel incorporation of discrete choice models and passive stochastic optimization into time-varying utility tracking. Experiments validate the framework’s effectiveness in detecting constrained utility-maximizing behavior, designing statistical detectors, and reverse-engineering dynamic decision-making systems.

Technology Category

Reasoning under Uncertainty: Stochastic OptimizationMachine Learning: Imitation Learning & Inverse Reinforcement LearningSearch and Optimization: Learning to Search

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingEconomics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

This monograph, spanning three chapters, explores Inverse Reinforcement Learning (IRL). The first two chapters view inverse reinforcement learning (IRL) through the lens of revealed preferences from microeconomics while the third chapter studies adaptive IRL via Langevin dynamics stochastic gradient algorithms. Chapter uses classical revealed preference theory (Afriat's theorem and extensions) to identify constrained utility maximizers based on observed agent actions. This allows for the reconstruction of set-valued estimates of an agent's utility. We illustrate this procedure by identifying the presence of a cognitive radar and reconstructing its utility function. The chapter also addresses the construction of a statistical detector for utility maximization behavior when agent actions are corrupted by noise. Chapter 2 studies Bayesian IRL. It investigates how an analyst can determine if an observed agent is a rationally inattentive Bayesian utility maximizer (i.e., simultaneously optimizing its utility and observation likelihood). The chapter discusses inverse stopping-time problems, focusing on reconstructing the continuation and stopping costs of a Bayesian agent operating over a random horizon. We then apply this IRL methodology to identify the presence of a Bayes-optimal sequential detector. Additionally, Chapter 2 provides a concise overview of discrete choice models, inverse Bayesian filtering, and inverse stochastic gradient algorithms for adaptive IRL. Finally, Chapter 3 introduces an adaptive IRL approach utilizing passive Langevin dynamics. This method aims to track time-varying utility functions given noisy and misspecified gradients. In essence, the adaptive IRL algorithms presented in Chapter 3 can be conceptualized as inverse stochastic gradient algorithms, as they learn the utility function in real-time while a stochastic gradient algorithm is in operation.

Problem

• Reconstruct agent's utility function from observed actions
• Detect Bayesian utility maximizer behavior in noisy settings
• Track time-varying utility functions with noisy gradients

Innovation

• Reconstructs utility functions via revealed preferences
• Uses Bayesian IRL for rational agent analysis
• Adapts Langevin dynamics for time-varying utilities