🤖 AI Summary
This study addresses the challenge of achieving high-probability safety in reinforcement learning under stochastic reach-avoid safety constraints. The work proposes an online algorithm that integrates entropy regularization into an optimistic framework for uncertainty (OFU) tailored to constrained Markov decision processes. Notably, this is the first approach to incorporate entropy regularization within the OFU paradigm, enabling strict safety guarantees throughout the learning process while substantially reducing inter-episode policy variability. Through finite-sample analysis, the authors derive a regret bound for the proposed algorithm and theoretically demonstrate that entropy regularization not only enhances performance but also effectively controls policy variance.
📝 Abstract
We consider the problem of learning the optimal policy for Markov decision processes with safety constraints. We formulate the problem in a reach-avoid setup. Our goal is to design online reinforcement learning algorithms that ensure safety constraints with arbitrarily high probability during the learning phase. To this end, we first propose an algorithm based on the optimism in the face of uncertainty (OFU) principle. Based on the first algorithm, we propose our main algorithm, which utilizes entropy regularization. We investigate the finite-sample analysis of both algorithms and derive their regret bounds. We demonstrate that the inclusion of entropy regularization improves the regret and drastically controls the episode-to-episode variability that is inherent in OFU-based safe RL algorithms.