Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions

2026-07-27

AI Summary

This work addresses the challenge in offline reinforcement learning where true actions are unobserved and only noisy proxy variables are available, leading to biased policy evaluation. To tackle this issue, the paper proposes the first offline reinforcement learning framework for settings with hidden actions. The approach leverages the next-state observation as a natural instrumental proxy for the latent action and integrates Markov decision process modeling with influence function techniques to construct the LURE estimator. This estimator enjoys multiple robustness properties and asymptotic normality, enabling both identifiability of the target policy’s value and valid statistical inference. Empirical evaluations on simulated environments and the MIMIC-III sepsis management dataset demonstrate that LURE substantially outperforms existing methods.

Abstract

Standard offline reinforcement learning (RL) algorithms typically assume that the actions in the dataset are observed without error. However, in many real-world applications, the true actions are unobserved and only noisy proxies are available, causing existing RL methods to yield biased and potentially misleading conclusions. We study off-policy evaluation in infinite-horizon discounted Markov decision processes with hidden actions. By leveraging the next-state variable as a natural proxy for the unobserved action, we establish identification of the policy value and propose an influence-function-based estimator called LURE (Learning from the Unseen: Robust Estimator). LURE is multiply robust, remaining consistent under several combinations of correctly specified nuisance components, and is asymptotically normal, enabling valid statistical inference. To our knowledge, this is the first work to address offline RL with hidden actions. We demonstrate LURE's effectiveness through simulations and a sepsis management application using the MIMIC-III database.

Problem

• offline reinforcement learning
• hidden actions
• off-policy evaluation
• Markov decision processes

Innovation

• offline reinforcement learning
• hidden actions
• off-policy evaluation
• multiply robust estimation
• influence function