🤖 AI Summary
This work addresses the high computational complexity and inefficiency of value function approximation in high-dimensional structured Markov decision processes (MDPs). By revealing the low-dimensional geometric structure of decision tessellations induced by optimal policies, the authors propose a boundary-driven policy approximation method that directly learns policy regions rather than value functions. They further introduce a policy loss decomposition mechanism that quantitatively links performance degradation to action boundary errors. Evaluated on inventory control and queue admission tasks, the proposed approach significantly reduces policy error and value gap compared to standard reinforcement learning baselines, achieving faster error convergence and enhanced training stability.
📝 Abstract
We study optimal-policy geometry in structured Markov decision processes. While approximate dynamic programming and reinforcement learning typically approximate high-dimensional value functions, we show that optimal policies induce simpler decision tessellations. We propose boundary-based policy approximations that learn policy regions directly. A policy-loss decomposition links performance degradation to action margins and explains why errors concentrate near indifference boundaries. Inventory control and queue admission experiments show lower policy error, smaller value gaps, faster error decay, and stability than reinforcement learning baselines.