Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning

2026-07-26

AI Summary

This work addresses the challenges of exploration and inefficient policy learning in sparse-reward, long-horizon tasks by proposing a two-level hierarchical reinforcement learning framework. The high-level controller performs strategic planning to guide long-term exploration, while the low-level policy leverages Soft Actor-Critic (SAC) for continuous control, augmented with entropy regularization to enhance both policy diversity and stability. By effectively integrating hierarchical structure with maximum-entropy learning, the proposed method significantly outperforms standard SAC baselines on the SAR-2 dataset, achieving notable improvements in task success rate, environmental coverage efficiency, and convergence speed.

Abstract

Exploration in sparse-reward long-horizon tasks poses significant challenges for reinforcement learning. To address these challenges, we propose a two-level Hierarchical Reinforcement Learning (HRL) framework. The first level handles high-level strategic planning, while the low-level uses the continuous-control Soft Actor-Critic (SAC) algorithm, and they utilize entropy-regularized policy optimization. The proposed framework was trained and evaluated using the Search-and-Rescue-2 (SAR-2) dataset. HRL-SAC effectively addresses sparse-reward long-horizon search problems characterized by delayed rewards and continuous control, and its outperforming the flat SAC baseline reinforcement learning in terms of success rates, coverage efficiency, and convergence. These findings indicate that hierarchical entropy-regularized policies are a promising solution to tackle long-horizon sparse-reward reinforcement learning tasks.

Problem

• sparse-reward
• long-horizon
• reinforcement learning
• exploration
• continuous control

Innovation

• Hierarchical Reinforcement Learning
• Soft Actor-Critic
• Sparse-Reward
• Long-Horizon
• Entropy-Regularized Policy