Quantifying Potential Observation Missingness in Inverse Reinforcement Learning

2026-05-12

AI Summary

This work addresses a critical yet often overlooked issue in inverse reinforcement learning (IRL): expert demonstrations may appear suboptimal due to missing observational information at the time of decision-making, leading standard IRL methods to infer biased reward functions. The paper presents the first systematic modeling and quantification of this inference bias caused by partial observability. It introduces a “minimal observation perturbation” framework that evaluates the impact of data incompleteness on reward learning by computing the smallest correction to the observed state needed to restore the expert’s policy as optimal. Grounded in IRL theory and optimization, the approach is validated on navigation tasks, simulated cancer treatment scenarios, and real-world ICU treatment data. Results demonstrate not only the misleading nature of conventional IRL under incomplete observations but also provide an interpretable metric for quantifying inference bias.

Abstract

Inverse reinforcement learning (IRL), which infers reward functions from demonstrations, is a valuable tool for modeling and understanding decision-making behavior. Many variants of IRL have been developed to capture complexities of human decision-making, such as subjective beliefs, imperfect planning, and dynamic goals. However, an often-overlooked issue in real-world behavioral datasets is that the recorded data may be missing observations that were available to the original decision-maker. In use-inspired settings such as healthcare, this can make expert actions appear suboptimal, even when they were near-optimal given the information available at the time. As a result, the rewards learned by standard IRL may be misleading. In this paper, we identify the minimal perturbations to the recorded observations needed for the expert's actions to appear optimal. We develop a practical algorithm for this problem and demonstrate its utility for quantifying the possible extent of missing observations in behavioral datasets through extensive experiments on synthetic navigation tasks, a cancer treatment simulator, and ICU treatment data.

Problem

• inverse reinforcement learning
• observation missingness
• behavioral data
• reward inference
• expert demonstrations

Innovation

• inverse reinforcement learning
• observation missingness
• minimal perturbations
• behavioral modeling
• reward inference