What Matters in Orchestrating Robot Policies: A Systematic Study of Hierarchical VLA Agents

2026-06-08

AI Summary

Existing hierarchical vision-language-action (Hi-VLA) systems lack a unified design principle, resulting in inconsistent approaches to task decomposition, module interfacing, and state representation. This work systematically integrates diverse Hi-VLA agents within an options-based control framework and evaluates design choices for high-level planners and low-level controllers—along with their interaction mechanisms—on a multitask benchmark encompassing short-horizon, long-horizon, and reasoning-intensive tasks, using both simulated and real-world ALOHA robots. The study distills practical design principles for constructing Hi-VLA systems, revealing the synergistic effects between model selection and interface mechanisms. The proposed approach significantly outperforms both flat VLA and naive hierarchical baselines, establishing a foundation for building more capable, robust, and principled hierarchical agents.

Abstract

Hierarchical vision-language-action (Hi-VLA) systems have emerged as a promising paradigm for complex robot manipulation, by using high-level VLM planners to decompose tasks into language subgoals executed by low-level VLA controllers. Despite recent empirical progress, there is a lack of unified design principles for these systems: existing Hi-VLA systems differ in how they choose and connect planners, controllers, mechanisms to switch between the two, and how observations and memory are represented in the planner. In this paper, we present a systematic study of Hi-VLA design for robot manipulation. We unify representative Hi-VLA agents under an options-style control framework and benchmark core design choices across short-horizon, long-horizon, and reasoning-intensive tasks. Our analysis distills practical principles for building Hi-VLA systems, showing how model choices and interface mechanisms jointly shape performance. Applying these principles yields a substantially stronger system than either flat VLA control or a naively designed hierarchy, across experiments both in simulation and on a real ALOHA robot. Overall, our results provide a foundation for building more capable, robust, and principled hierarchical VLA agents. More information and video at jiahenghu.github.io/hi-vla.

Problem

• Hierarchical VLA
• robot manipulation
• design principles
• vision-language-action
• systematic study

Innovation

• Hierarchical VLA
• systematic study
• options framework
• robot manipulation
• vision-language-action