🤖 AI Summary
This work studies the global convergence of the Fisher–Rao policy gradient flow for infinite-horizon entropy-regularized Markov decision processes (MDPs) on Polish spaces. Addressing the non-convex policy optimization objective, we establish, for the first time, the global well-posedness and exponential convergence of this gradient flow under the Fisher–Rao metric, along with its robustness to policy gradient estimation error. Our analysis integrates the performance difference lemma, dual-flow techniques, and entropy-regularized MDP theory to uncover intrinsic connections between the Fisher–Rao gradient flow, mirror descent, and natural policy gradients. The results provide the first strong convergence guarantee—under non-convexity—for discrete-time policy gradient algorithms such as soft Q-learning and natural policy gradient. Moreover, they fill a fundamental gap in the global convergence analysis of continuous-time policy flows over general Riemannian metric spaces.
📝 Abstract
We study the global convergence of a Fisher-Rao policy gradient flow for infinite-horizon entropy-regularised Markov decision processes with Polish state and action space. The flow is a continuous-time analogue of a policy mirror descent method. We establish the global well-posedness of the gradient flow and demonstrate its exponential convergence to the optimal policy. Moreover, we prove the flow is stable with respect to gradient evaluation, offering insights into the performance of a natural policy gradient flow with log-linear policy parameterisation. To overcome challenges stemming from the lack of the convexity of the objective function and the discontinuity arising from the entropy regulariser, we leverage the performance difference lemma and the duality relationship between the gradient and mirror descent flows. Our analysis provides a theoretical foundation for developing various discrete policy gradient algorithms.
