🤖 AI Summary
In complex continuous multi-task environments, planning models are difficult to obtain, trial-and-error learning suffers from low sample efficiency, and existing hierarchical RL methods are restricted to discrete settings and lack cross-task generalization. Method: We propose a hierarchical reinforcement learning framework that integrates expert-provided abstractions. It dynamically compiles high-fidelity human task abstractions into subgoal generators, constructs goal-conditioned policies, and performs sparse reward shaping using the optimal state-value function of an abstract MDP. Contribution/Results: This work achieves the first seamless integration of expert abstractions into hierarchical RL, relaxing longstanding assumptions of discrete abstraction and task-specificity. Evaluated on procedurally generated continuous control benchmarks, our approach significantly improves sample efficiency and task success rates, scales to more complex tasks, and enables zero-shot generalization to unseen scenarios.
📝 Abstract
Decision-making in complex, continuous multi-task environments is often hindered by the difficulty of obtaining accurate models for planning and the inefficiency of learning purely from trial and error. While precise environment dynamics may be hard to specify, human experts can often provide high-fidelity abstractions that capture the essential high-level structure of a task and user preferences in the target environment. Existing hierarchical approaches often target discrete settings and do not generalize across tasks. We propose a hierarchical reinforcement learning approach that addresses these limitations by dynamically planning over the expert-specified abstraction to generate subgoals to learn a goal-conditioned policy. To overcome the challenges of learning under sparse rewards, we shape the reward based on the optimal state value in the abstract model. This structured decision-making process enhances sample efficiency and facilitates zero-shot generalization. Our empirical evaluation on a suite of procedurally generated continuous control environments demonstrates that our approach outperforms existing hierarchical reinforcement learning methods in terms of sample efficiency, task completion rate, scalability to complex tasks, and generalization to novel scenarios.