Linear Programming Representations and Strongly Polynomial Algorithms for Robust Markov Decision Processes

2026-10-01

AI Summary

This study addresses the efficient solution and complexity analysis of robust Markov decision processes (RMDPs). It proposes a general complexity analysis framework based on encoded policy iteration, which reduces RMDPs to a single linear program (LP) and achieves strongly polynomial-time solvability through polyhedral uncertainty modeling. The key contributions include deriving exact LP representations for optimal values and policies, substantially tightening complexity bounds under interval and Wasserstein uncertainty sets to establish new strongly polynomial algorithmic limits, and extending these theoretical results to stochastic game settings.

Abstract

We study linear programming (LP) representations and strongly polynomial algorithms for robust Markov decision processes (RMDPs) with rational polyhedral state-action rectangular uncertainty in rewards and transitions. By encoding a finite sequence of robust policy-iteration steps, we construct a single LP whose optimal solutions recover the robust optimal value and all optimal stationary randomized policies. At fixed discount, the LP has polynomial dimension and encoding length and can be constructed in strongly polynomial time. We also develop a general complexity analysis of robust policy iteration that combines the cost of minimizing over uncertainty sets with the number of iterations needed to evaluate a policy. For a fixed discount factor, we use this analysis to improve the known complexity bounds for $\ell_1$ and $\ell_\infty$ RMDPs and establish new strongly polynomial bounds for general interval, weighted $\ell_1$, and Wasserstein RMDPs, as well as turn-based stochastic games with these uncertainty sets.

Problem

• Robust Markov Decision Processes
• Linear Programming
• Strongly Polynomial Algorithms
• Uncertainty Sets
• Complexity Bounds

Innovation

• Robust Markov Decision Processes
• Linear Programming
• Strongly Polynomial Algorithms
• Policy Iteration
• Complexity Bounds