🤖 AI Summary
This paper addresses policy optimization for discounted infinite-horizon constrained Markov decision processes (CMDPs) in online learning for safety-critical systems, aiming to maximize expected cumulative reward while *strictly satisfying* cumulative constraints throughout the entire learning process. We propose the first model-free and simulation-free interior-point method framework, which guarantees policy feasibility at *every training iteration*—not merely asymptotically. Our approach constructs an interior-point regularized objective using a log-barrier function and integrates it with policy gradient updates under a Fisher non-degeneracy assumption on policy parameterization. Theoretically, we establish a sample complexity of $ ilde{mathcal{O}}(varepsilon^{-6})$ for converging to an $varepsilon$-optimal feasible policy. This incurs only an $mathcal{O}(varepsilon^{-2})$ overhead relative to the unconstrained C-NPG-PDA algorithm, significantly improving both learning efficiency and reliability under safety constraints.
📝 Abstract
We consider discounted infinite horizon constrained Markov decision processes (CMDPs) where the goal is to find an optimal policy that maximizes the expected cumulative reward subject to expected cumulative constraints. Motivated by the application of CMDPs in online learning of safety-critical systems, we focus on developing a model-free and simulator-free algorithm that ensures constraint satisfaction during learning. To this end, we develop an interior point approach based on the log barrier function of the CMDP. Under the commonly assumed conditions of Fisher non-degeneracy and bounded transfer error of the policy parameterization, we establish the theoretical properties of the algorithm. In particular, in contrast to existing CMDP approaches that ensure policy feasibility only upon convergence, our algorithm guarantees the feasibility of the policies during the learning process and converges to the $varepsilon$-optimal policy with a sample complexity of $ ilde{mathcal{O}}(varepsilon^{-6})$. In comparison to the state-of-the-art policy gradient-based algorithm, C-NPG-PDA, our algorithm requires an additional $mathcal{O}(varepsilon^{-2})$ samples to ensure policy feasibility during learning with the same Fisher non-degenerate parameterization.
