Non-Asymptotic Best Policy Identification Guarantees in Online Reinforcement Learning

2026-07-19

AI Summary

This work addresses the problem of identifying an optimal policy in tabular Markov decision processes (MDPs) with high confidence under online reinforcement learning, aiming to minimize the expected sample complexity. The authors adopt an active sequential hypothesis testing framework and propose the Navigate and Stop algorithm, which achieves efficient exploration and verification under deterministic rewards. They establish the first non-asymptotic upper bound on the sample complexity of this algorithm, explicitly quantifying the influence of instance-specific structural properties—such as the characteristic time and MDP connectivity—on the required number of samples. This result fills a notable theoretical gap by providing a rigorous, finite-sample analysis that elucidates how intrinsic problem structure governs learning efficiency in best-policy identification.

Abstract

In this work we study the Best Policy Identification (BPI) problem in online, tabular Reinforcement Learning. This is an active sequential hypothesis testing problem in which the learner's objective is to identify an optimal policy in a Markov Decision Process (MDP) with high confidence, while minimizing the expected sample complexity to do so. We consider an online setting with deterministic rewards, where the agent must strategically navigate through the MDP in order to effectively explore. Previous works in the literature have provided asymptotically optimal methods for BPI, such as the Navigate and Stop (NaS) algorithm and its variants, however existing analysis remains asymptotic. In this work, we fill that gap by providing the first non-asymptotic sample complexity guarantees for NaS, showing that its sample complexity depends not only on the characteristic time, but also on the connectivity of the underlying MDP, the curvature of the optimal characteristic time, and other instance-dependent quantities. We identify these additional attributes and make explicit their contributions to the overall sample complexity.

Problem

• Best Policy Identification
• Online Reinforcement Learning
• Sample Complexity
• Markov Decision Process
• Non-Asymptotic Guarantees

Innovation

• non-asymptotic analysis
• best policy identification
• sample complexity
• online reinforcement learning
• Markov Decision Process