Finite-Time Analysis of the Natural Policy Gradient in Finite-Horizon Markov Decision Processes

2026-07-24

AI Summary

This work addresses the lack of finite-time convergence guarantees for natural policy gradient (NPG) algorithms in finite-horizon Markov decision processes (MDPs). Under the setting where the dynamics are known and the transition kernel is time-dependent, the paper provides the first non-asymptotic convergence analysis for exact NPG. By introducing a linear MDP projection oracle, the authors establish that with a constant stepsize, the algorithm converges at a sublinear rate of $O(H^2/t)$, while an exponentially increasing stepsize $\eta_t = \eta_0 (H/(H-1))^t$ yields a linear convergence rate of $O((1 - 1/\vartheta_\rho)^t)$. This study bridges a critical theoretical gap in finite-horizon NPG analysis and highlights the pivotal role of stepsize scheduling in determining convergence speed.

Abstract

Natural Policy Gradient (NPG) is a well-established Reinforcement Learning algorithm that underlies widely used methods such as Trust Region Policy Optimization and Proximal Policy Optimization, both of which have demonstrated strong empirical success. In this paper, we study exact NPG in finite-horizon Markov Decision Processes with known dynamics and horizon-dependent transition kernels. We provide the first finite-time convergence guarantees for this algorithm in this setting, for which we consider both constant and increasing step size regimes. With a constant step size $η_t=η$, we prove that NPG converges sublinearly with a rate of $\mathcal{O}(H^{2}/t)$ after $t$ iterations, where $H$ is the horizon length. We also extend this constant step size analysis to linear MDPs in an exact population-projection oracle under a full support projection distribution, recovering the same sublinear rate as in the tabular setting. Furthermore, with increasing step sizes, we prove that this algorithm achieves a linear convergence rate of $\mathcal{O}\left(\left(1-\frac{1}{\vartheta_ρ}\right)^t\right)$ for a problem-dependent constant $\vartheta_ρ> 1$, and the horizon-only robust schedule of the form $η_t=η_0(H/(H-1))^t$ where $η_0>0$ and $H \geq 2$, attains this same geometric rate.

Problem

• Natural Policy Gradient
• Finite-Horizon MDP
• Finite-Time Convergence
• Reinforcement Learning
• Markov Decision Processes

Innovation

• Natural Policy Gradient
• Finite-Horizon MDPs
• Finite-Time Convergence
• Linear Convergence Rate
• Step Size Schedule