Addressing the Orchestration Gap in Generalist Robots via Physical Agency

2026-07-23

AI Summary

General-purpose robots struggle to jointly perceive, plan, execute, and recover in complex tasks, as existing end-to-end policies are limited by insufficient reasoning and coordination capabilities. This work proposes Pigey, a physically grounded agent orchestration framework that formally defines and bridges the “orchestration gap” by decoupling high-level task planning from low-level vision-language-action (VLA) policies. Pigey establishes a closed-loop control system that enables goal decomposition, instruction scheduling, outcome verification, and failure recovery. Notably, it achieves substantial performance gains on frozen VLA policies for complex reasoning tasks without requiring additional data or fine-tuning. On the LIBERO-PRO benchmark, the success rate improves from 12.8% to 53.3%, and real-world robotic task success rises from near zero to over 90%.

Abstract

General-purpose robots need to reason about their actions, combining perception, world knowledge, planning, success detection, recovery, and low-level control. Today's state-of-the-art models attempt to combine all these capabilities into the learned policy via large-scale pre-training. Instead, we show that these capabilities can be decomposed into a general language-conditioned policy/control agent and a high-level agent manager/orchestrator. Rather than training policies to reason via pre-training, we build a closed-loop physical agent orchestrator that can do high-level planning, decompose the goal into achievable subgoals, command low-level motor commands, track and verify the outcome from low-level observations, and recover from failures. Our Physical Agency orchestrator (Pigey) can control existing vision-language-action (VLA) policies as well as parametrized skills to solve complex reasoning tasks in the real world, without any additional data collection or post-training. We evaluate Pigey extensively across simulation benchmarks and challenging real-world robotic manipulation tasks, and demonstrate significant performance improvements over existing generalist policies. On LIBERO-PRO, Pigey advances the state-of-the-art by over 4x (12.8% -> 53.3%) with no task-specific fine-tuning. On a real robot, Pigey lifts the frozen policy from near-zero to over 90% on reasoning-limited tasks. We call the difference between what frozen motor skills achieve alone and inside the agentic loop the orchestration gap.

Problem

• orchestration gap
• generalist robots
• physical agency
• reasoning tasks
• robotic manipulation

Innovation

• Physical Agency
• Orchestrator
• Generalist Robots
• Vision-Language-Action Policies
• Orchestration Gap