Vector Bellman Theory for Multichain Robust Average-Reward Markov Decision Processes

2026-09-23

AI Summary

This study addresses the challenge of optimizing long-term performance in multichain robust average-reward Markov decision processes, where initial-state dependence complicates such optimization. To overcome this, the work develops a vector Bellman theory that constructs a coupled system prioritizing gain over bias. Uncertainty is modeled using compact rectangular ambiguity sets, and by integrating finite Bellman certificates with undiscounted planning techniques, a robust approximate shifted Halpern algorithm based on asymptotic affine trajectories is proposed. The authors prove that gain estimates converge to optimal values and that the resulting greedy controller achieves average optimality under finite budgets. Crucially, this approach enables identifying the optimal robust gain from arbitrary initial states while generating stationary saddle-point policies, thereby completing the associated theoretical framework for multichain robust MDPs.

Abstract

Robust average-reward Markov decision processes provide a fundamental framework for long-term performance optimization under uncertainty, and can have optimal long-run rewards that depend on the initial state. This state dependence requires a vector Bellman theory that accounts for both recurrent-class rewards and transition uncertainty. We develop such a theory for finite models with compact, post-action $(s,a)$-rectangular ambiguity. A gain-first, bias-second optimization principle yields a coupled vector gain-bias system, and every finite solution identifies the optimal robust gain and supplies stationary saddle strategies against history-dependent opponents, simultaneously from all initial states. We further characterize solvability through stationary gain conditions and a uniform bound on canonical transient corrections, and give sufficient conditions that permit distinct recurrent-class gains. The certificates also yield asymptotically affine trajectories of the robust Bellman operator, based on which we design a robust approximately shifted Halpern planning algorithm. Under finite Bellman solvability, the gain estimates and Bellman displacements converge to the optimal gain vector, and every extracted greedy controller is average-optimal after a finite, instance-dependent budget. These results thus connect finite Bellman certificates to undiscounted planning for state-dependent robust average rewards, providing theoretical understandings.

Problem

• Robust Markov Decision Processes
• Average-Reward
• Vector Bellman Theory
• Multichain
• Uncertainty

Innovation

• Vector Bellman Theory
• Robust Average-Reward MDPs
• Multichain
• Gain-Bias System
• Halpern Planning Algorithm