train hierarchical rl controllers

Existing hierarchical decision-making approaches often struggle to simultaneously satisfy constraints and maintain computational efficiency due to misalignment between low-level policies and high-level objectives. This work proposes a principled inverse optimization–based hierarchical framework that, for the first time, systematically constructs structured low-level optimization problems from expert demonstrations, thereby aligning high-level task abstractions with low-level decision-making. By integrating inverse optimization, hierarchical reinforcement learning, and optimal control, the method achieves both interpretability and computational efficiency. Empirical evaluations on resource allocation and obstacle avoidance tasks demonstrate that the approach significantly outperforms end-to-end reinforcement learning, learning-augmented optimal control, and existing hierarchical methods, achieving state-of-the-art performance in both decision quality and computational speed.

In complex continuous multi-task environments, planning models are difficult to obtain, trial-and-error learning suffers from low sample efficiency, and existing hierarchical RL methods are restricted to discrete settings and lack cross-task generalization. Method: We propose a hierarchical reinforcement learning framework that integrates expert-provided abstractions. It dynamically compiles high-fidelity human task abstractions into subgoal generators, constructs goal-conditioned policies, and performs sparse reward shaping using the optimal state-value function of an abstract MDP. Contribution/Results: This work achieves the first seamless integration of expert abstractions into hierarchical RL, relaxing longstanding assumptions of discrete abstraction and task-specificity. Evaluated on procedurally generated continuous control benchmarks, our approach significantly improves sample efficiency and task success rates, scales to more complex tasks, and enables zero-shot generalization to unseen scenarios.

This paper addresses hierarchical control in a two-level environment: a high-level known graph (“map”) whose vertices represent unknown, dynamically evolving MDPs (“rooms”). We propose an end-to-end controller framework integrating deep reinforcement learning (DRL) with reactive synthesis. Methodologically, the low level trains reusable latent policies within each room using PAC learning theory—avoiding model distillation and improving robustness to sparse rewards; the high level employs reactive synthesis to generate a dynamic scheduler satisfying Linear Temporal Logic (LTL) specifications. Theoretically, we establish the first PAC performance guarantees for hierarchical policies and derive bounds on abstraction quality. Experimentally, on navigation tasks with dynamic obstacles, our framework significantly improves policy generalization across rooms and enhances reliability of high-level scheduling decisions.

This paper addresses the absence of a standardized taxonomy for hierarchical multi-agent systems (HMAS) and the lack of clarity in trade-offs among coordination mechanisms. To this end, we propose the first multidimensional classification framework integrating structural, temporal, and communication dimensions—comprising five axes: control hierarchy, information flow, task allocation, temporal layering, and communication topology. The framework systematically unifies classical coordination paradigms (e.g., Contract Net), hierarchical reinforcement learning, and large language model–based agents, thereby exposing novel challenges in interpretability, scalability, and secure integration. Empirical validation in industrial domains—including power grid management and oilfield operations—demonstrates that the framework effectively guides HMAS design, enhancing global efficiency while preserving local autonomy. It exhibits strong cross-domain adaptability and practical utility for real-world system engineering.

This work addresses performance bottlenecks in multi-agent reinforcement learning (MARL) arising from sparse rewards, high-dimensional state-action spaces, and the challenge of coordinated policy learning. The authors propose a hierarchical architecture wherein a pretrained large language model (LLM) serves as a centralized strategic controller at the high level, dynamically selecting among specialized low-level reinforcement learning policies without relying on handcrafted rules. This approach represents the first integration of LLMs into high-level planning for multi-agent systems, significantly enhancing tactical diversity and behavioral human-likeness. Evaluated on a 2v2 capture-the-flag task, the method achieves a win rate of 46.4%, matching the performance of hand-designed behavior trees and substantially outperforming flat RL baselines. A user study further reveals that 60% of participants judged the agents’ behavior as most human-like (p = 0.027).

This work addresses the instability in high-level subgoal selection within hierarchical reinforcement learning, which arises due to sparse and delayed environmental feedback and is exacerbated by the limitations of low-level execution capabilities. To mitigate this issue, the authors propose an intrinsic motivation mechanism based on coarse-grained dynamic modeling: a coarse dynamics model is constructed by aggregating multi-step environmental transitions, and a Mixture Density Network (MDN) is employed to quantify the predictive uncertainty of this model. This uncertainty is then used as a risk-sensitive intrinsic reward to guide the high-level agent away from subgoals associated with high uncertainty. Evaluated on non-stationary, long-horizon tasks, the proposed method significantly outperforms existing hierarchical reinforcement learning algorithms, demonstrating improved task completion efficiency and policy stability.

In dynamic, highly constrained environments, end-to-end multi-agent cooperative control suffers from low sample efficiency and poor reliability, while model-based approaches exhibit limited generalization. To address these challenges, this paper proposes a hierarchical reinforcement learning framework: a high-level RL policy performs structured Region-of-Interest (ROI)-guided tactical decision-making, while a low-level Model Predictive Control (MPC) module executes safe motion planning. Our key innovation lies in the explicit coupling of ROI-driven target selection with MPC-based execution—enabling behavioral generalization without predefined reference trajectories. Evaluated on a predator–prey benchmark task, the method achieves significant improvements over end-to-end and masked RL baselines: +23.6% in cumulative reward, −58.4% in collision rate (enhancing safety), and improved group behavioral consistency.

This work addresses the limited policy expressivity and insufficient exploration efficiency in hierarchical reinforcement learning by proposing a novel paradigm in which a controller constructs a flexible behavioral space through linear combinations of multiple option reward functions. Departing from conventional assumptions of long-horizon planning, the approach demonstrates that the core advantage of hierarchical structures lies in their capacity to enhance exploration. Empirical evaluation in the NetHack Learning Environment shows that the proposed method substantially improves both exploration efficiency and overall learning performance, thereby validating its effectiveness and superiority in complex environments.

This work proposes a semantics-preserving hierarchical compression framework for Markov Decision Processes (MDPs) to address sequential decision-making problems with inherent multi-level structures. By abstracting families of low-level policies into atomic actions within a higher-level MDP, the approach decouples subtasks and substantially reduces the policy search space. Leveraging skill embeddings and higher-order functional decomposition, the framework integrates skill-based curriculum learning with meta-reinforcement learning to enable effective cross-task and cross-hierarchy skill transfer. Experimental results in environments such as MazeBase+ demonstrate that the method significantly enhances abstraction capability and curriculum learning efficiency, markedly decreasing both the number of iterations and computational overhead required to solve the MDP.