Hierarchical Multilevel Monte Carlo for Order-Optimal Neural Actor-Critic in Average-Reward CMDPs

2026-07-30

AI Summary

This work addresses a fundamental bias–variance trade-off in neural critics within average-reward constrained Markov decision processes (CMDPs), which hinders primal-dual methods from achieving minimax optimal convergence. To overcome this, the authors propose a hierarchical multilevel Monte Carlo (MLMC) neural critic that simultaneously debiases trajectory sampling and critic optimization, yielding low-bias value estimates at only logarithmic expected sample cost. Building upon this, they develop a primal-dual natural actor-critic algorithm and establish, for the first time, minimax optimal convergence guarantees in infinite-horizon average-reward CMDPs under general policy parameterizations and neural critics—without requiring assumptions on the system’s mixing time. Notably, these guarantees also advance the unconstrained setting. The algorithm achieves both objective suboptimality gap and constraint violation of order $\tilde{O}(T^{-1/2})$, matching the theoretical lower bound.

Abstract

Constrained Markov Decision Processes (CMDPs) provide a natural framework for reinforcement learning in safety-critical applications, where agents maximize long-term reward while satisfying long-term constraints. Although primal-dual actor-critic methods with linear critics are well understood, extending order-optimal convergence guarantees to neural critics in average-reward CMDPs has remained open. The main challenge is a fundamental bias-cost trade-off in neural critic estimation: under Neural Tangent Kernel (NTK) analysis, reducing critic bias substantially increases critic optimization cost, preventing order-optimal convergence in the primal-dual framework. We resolve this bottleneck by introducing a hierarchical Multilevel Monte Carlo (MLMC) neural critic that performs debiasing simultaneously across trajectory sampling and critic optimization. The resulting estimator attains the bias of a long critic optimization run with only logarithmic expected sample cost. Building on this estimator, we develop a primal-dual Natural Actor-Critic algorithm that achieves both an optimality gap and a constraint violation of order $\tilde{O}(T^{-1/2})$. This establishes the first order-optimal convergence guarantees for infinite-horizon average-reward CMDPs with general policy parameterization and neural critics, while eliminating the need to know the underlying mixing time. Our results are novel even in the unconstrained setting.

Problem

• Constrained Markov Decision Processes
• Neural Actor-Critic
• Average-Reward
• Order-Optimal Convergence
• Bias-Cost Trade-off

Innovation

• Hierarchical Multilevel Monte Carlo
• Neural Actor-Critic
• Average-Reward CMDPs
• Order-Optimal Convergence
• Bias-Cost Trade-off