Instance-Dependent Regret for CMDPs with Step-Wise Constraints

2026-10-01

AI Summary

This study addresses the online learning problem in Markov Decision Processes with step-size constraints by proposing the SVAE algorithm, which achieves efficient learning through exploring candidate safe subgraph structures and performing variance-adaptive optimistic planning. Theoretically, it establishes for the first time an instance-dependent regret lower bound that relies on specific variances, proving this dependency is unavoidable and thereby transcending traditional uniform complexity limitations. Methodologically, the approach integrates variance-adaptive exploration with statistical complexity analysis to handle constraint satisfaction. Ultimately, the proposed algorithm attains instance-dependent sublinear cumulative regret bounds along with polylogarithmic violation bounds, significantly outperforming worst-case baselines.

Abstract

We study online learning in episodic tabular constrained Markov decision processes with step-wise safety constraints. In such a setting, the constraints induce a safe subgraph that shapes the variance of cumulative rewards under feasible policies and, consequently, the difficulty of learning. Exploiting this structure, however, requires learning which actions are safe while controlling constraint violations. We propose Safe Variance-Adaptive Exploration (SVAE), an efficient algorithm that learns candidate safe subgraphs and performs variance-adaptive optimistic planning within them. With high probability, SVAE achieves cumulative regret of order $\widetilde{\mathcal{O}}(\sqrt{SAH\min\{\mathbb{V}_Σ,K\mathrm{Var}^{\star}\}}+S\sqrt{AH^3\min\{K,\mathcal{C}\}}+S^2AH^2)$ over $K$ episodes, where $H$ is the horizon of a single episode, while $S$ and $A$ are the numbers of states and actions, respectively. Here, $\mathrm{Var}^{\star}$ is the maximum return variance among safe policies, $\mathbb{V}_Σ$ is the variance accumulated before the first unsafe action is encountered, and $\mathcal{C}$ captures the statistical complexity of eliminating actions incorrectly considered potentially safe. SVAE additionally attains $\widetilde{\mathcal{O}}(H\sqrt{SAK}+S^2AH^2)$ step-wise constraint violation and a gap-dependent violation bound that is polylogarithmic in $K$. Finally, we establish a lower bound showing that dependence on these instance-specific quantities is unavoidable.

Problem

• Constrained Markov Decision Processes
• Instance-Dependent Regret
• Step-wise Safety Constraints
• Online Learning

Innovation

• Constrained Markov Decision Processes
• Safe Variance-Adaptive Exploration
• Instance-Dependent Regret
• Step-Wise Constraints
• Optimistic Planning