Sample Efficient Hierarchical Reinforcement Learning via Best Policy Identification

2026-07-31

AI Summary

This work addresses the low sample efficiency of hierarchical reinforcement learning under sparse rewards by proposing the HBPI-UCRL algorithm, which establishes, for the first time, learnability conditions for parallel hierarchical reinforcement learning. Built upon a model-based framework, the method integrates optimal policy identification with UCRL (Upper Confidence Reinforcement Learning) and leverages a hierarchical structure wherein high-level policies induce multi-step transitions at the lower level. Under mild assumptions on the underlying dynamics of the lower level, HBPI-UCRL achieves polynomial sample complexity. Theoretical analysis demonstrates that its sample complexity in goal-oriented tasks is strictly superior to that of non-hierarchical approaches, thereby providing rigorous theoretical support for efficient hierarchical learning.

Abstract

We present HBPI-UCRL, a model-based algorithm for hierarchical reinforcement learning (HRL) that learns high-level and low-level policies in parallel. HBPI-UCRL exploits the fact that a high-level transition corresponds to a multi-step transition at the low level. We introduce two conditions on the low-level dynamics that are sufficient to make parallel HRL learnable. When these conditions hold, we prove that HBPI-UCRL has a polynomial sample complexity in the problem parameters. In the sparse-reward, goal-directed setting, our sample complexity upper bound for HBPI-UCRL is strictly lower than that of its non-hierarchical counterpart, providing theoretical justification for the empirical success of HRL.

Problem

• hierarchical reinforcement learning
• sample efficiency
• sparse reward
• goal-directed learning
• policy learning

Innovation

• Hierarchical Reinforcement Learning
• Sample Complexity
• Model-based RL
• Parallel Policy Learning
• Best Policy Identification