Beyond Flat Policies: Hierarchical Post-Training for Embodied Agents in Robotic Manipulation

2026-08-06

AI Summary

Existing vision-language-action (VLA) models typically employ flat post-training strategies that struggle to capture task progression, resulting in poor robustness for long-horizon manipulation. Hierarchical approaches, while promising, often rely on offline demonstrations and lack online learning capabilities. This work proposes HiRoC, a hierarchical robot control framework that decouples task planning from action execution: a high-level planner generates semantic subgoals, while a low-level executor optimizes subgoal-conditioned policies via reinforcement learning. To mitigate distributional mismatch between planner and executor, HiRoC aligns their subgoal distributions prior to training. As the first reinforcement learning–based hierarchical post-training framework, HiRoC significantly outperforms strong baselines and demonstrates its effectiveness and component contributions across multiple robotic manipulation benchmarks.

Abstract

Vision-language-action (VLA) models have demonstrated remarkable capabilities in robotic manipulation by leveraging pretrained vision-language models. However, existing post-training methods predominantly optimize VLA models as flat policies, making it difficult to explicitly model task progression and perform robust long-horizon manipulation. Although hierarchical approaches introduce task decomposition, they mainly rely on supervised learning from offline demonstrations and cannot effectively improve execution through online interaction. To address this limitation, we propose Hierarchical Robotic Control (HiRoC), a hierarchical post-training framework that decouples high-level task planning from low-level action execution. The planner decomposes complex tasks into executable subgoals to provide explicit semantic guidance, while the executor continuously improves subgoal-conditioned action generation through reinforcement learning. To enable effective collaboration between the two modules, we further align the executor with planner-generated subgoals before reinforcement learning, mitigating the distribution misalignment between planning and execution. Extensive experiments across diverse robotic manipulation benchmarks demonstrate that HiRoC consistently outperforms strong baselines. Comprehensive analyses further validate the effectiveness of hierarchical post-training and the contribution of each key component.

Problem

• hierarchical policy
• robotic manipulation
• post-training
• long-horizon tasks
• vision-language-action models

Innovation

• hierarchical reinforcement learning
• vision-language-action models
• task decomposition
• post-training alignment
• robotic manipulation