Online Policy Evaluation for MDPs with Dynamic UBSR Measures

2026-07-25

AI Summary

This work addresses the limitations of existing risk-aware reinforcement learning methods for policy evaluation, which are often confined to specific risk measures or rely on simulators, thereby hindering their applicability in fully online settings. Focusing on Markov decision processes under the dynamic utility-based shortfall risk (UBSR) measure, the study introduces UBSR-TD—an efficient online policy evaluation algorithm based on linear function approximation—by extending the risk-neutral temporal difference (TD) algorithm through a tailored loss function, along with an accelerated variant. Theoretical analysis establishes its almost sure convergence, while numerical experiments confirm its empirical effectiveness. Furthermore, the approach demonstrates practical utility in managing perishable inventory with uncertain shelf life, thereby overcoming key applicability barriers in online risk-aware policy evaluation.

Abstract

Developing efficient function-approximation methods for policy evaluation is a fundamental challenge in risk-aware reinforcement learning. Existing approaches either focus on restrictive classes of risk measures or rely on access to a simulator, limiting their applicability in fully online settings. In this work, we propose computationally efficient online learning algorithms for policy evaluation in Markov decision processes (MDPs) with dynamic utility-based shortfall risk (UBSR) measures under linear function approximation. Specifically, we introduce the UBSR-TD algorithm, establish conditions under which it converges almost surely, and develop several variants designed to accelerate convergence. Our formulation shows that existing policy evaluation algorithms for risk-neutral MDPs can be readily adapted to dynamic UBSR settings by incorporating a loss function into the temporal-difference error. Numerical experiments support our theoretical findings, and an application to a perishable inventory management problem with shelf-life uncertainty demonstrates the practical effectiveness of the proposed methods.

Problem

• policy evaluation
• risk-aware reinforcement learning
• dynamic UBSR measures
• online learning
• Markov decision processes

Innovation

• dynamic UBSR
• online policy evaluation
• risk-aware reinforcement learning
• temporal-difference learning
• linear function approximation