Distributional Off-Policy Evaluation with Deep Quantile Process Regression

2026-04-20

AI Summary

This work addresses a key limitation in conventional off-policy evaluation, which focuses solely on the expected return while ignoring the full distributional information of returns. To overcome this, we propose the DQPOPE algorithm, which introduces continuous quantile functions into off-policy evaluation for the first time, modeling the complete return distribution via deep quantile process regression. By integrating distributional reinforcement learning with deep neural networks, our method not only provides a more comprehensive characterization of policy value but also achieves significantly higher estimation accuracy and robustness under the same sample budget. Furthermore, we present a theoretical analysis of sample complexity grounded in deep network function approximation, establishing the statistical advantages and theoretical guarantees of the proposed distributional approach.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Distributed Search
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper investigates the off-policy evaluation (OPE) problem from a distributional perspective. Rather than focusing solely on the expectation of the total return, as in most existing OPE methods, we aim to estimate the entire return distribution. To this end, we introduce a quantile-based approach for OPE using deep quantile process regression, presenting a novel algorithm called Deep Quantile Process regression-based Off-Policy Evaluation (DQPOPE). We provide new theoretical insights into the deep quantile process regression technique, extending existing approaches that estimate discrete quantiles to estimate a continuous quantile function. A key contribution of our work is the rigorous sample complexity analysis for distributional OPE with deep neural networks, bridging theoretical analysis with practical algorithmic implementations. We show that DQPOPE achieves statistical advantages by estimating the full return distribution using the same sample size required to estimate a single policy value using conventional methods. Empirical studies further show that DQPOPE provides significantly more precise and robust policy value estimates than standard methods, thereby enhancing the practical applicability and effectiveness of distributional reinforcement learning approaches.

Problem

• off-policy evaluation
• distributional reinforcement learning
• return distribution
• quantile estimation

Innovation

• distributional off-policy evaluation
• deep quantile process regression
• continuous quantile function
• sample complexity analysis
• return distribution estimation