RoboHarness: Memory-Driven Orchestration of Heterogeneous Robot Policies for Long-Horizon Planning

2026-07-20

AI Summary

Long-horizon robotic tasks require the coordination of diverse heterogeneous policies, yet existing approaches struggle with ambiguous capability boundaries and distributional mismatches among them. This work proposes RoboHarness, a framework that encapsulates independently developed control policies—such as vision-language-action (VLA) models, reinforcement learning agents, and task and motion planning (TAMP) systems—as reusable skills. RoboHarness enables capability-aware task decomposition and policy routing through multimodal execution memory and online evidence, while a novel Memory Bridge mechanism stabilizes policy switching. Notably, this framework achieves the first orchestration of heterogeneous policies without joint retraining, offering generality and zero-shot generalization. Evaluated across three public benchmarks, 500 custom tasks, and 135 real-world trials, RoboHarness significantly improves long-horizon planning success rates and out-of-distribution robustness.

Abstract

Long-horizon robotic tasks require diverse capabilities that no single policy can reliably provide. Heterogeneous policies offer complementary strengths, but orchestrating them requires reasoning over uncertain capability boundaries and cross-policy distribution mismatch, which are largely overlooked by existing planning methods built on homogeneous, predefined skills with fixed applicability. We propose RoboHarness, a unified framework that encapsulates independently developed robot control systems as reusable agentic skills. Although instantiated in this work with VLAs, RL policies, and task-and-motion planning (TAMP) systems, RoboHarness is designed as a general framework compatible with a broader range of robot policies, such as navigation policies, model predictive controllers, and world-action models. RoboHarness uses multi-modal execution memory and online evidence to characterize policy capability boundaries for capability-aware decomposition and routing. To stabilize policy handoffs, its Memory Bridge retrieves execution trajectories associated with the next policy, estimates its in-distribution state region, and guides the robot toward that region without joint policy retraining. Extensive experiments on three public benchmarks, 500 customized tasks, and 135 real-robot experiments demonstrate effective capability-aware routing and stable policy orchestration, yielding substantial improvements in zero-shot long-horizon planning and out-of-distribution robustness.

Problem

• long-horizon planning
• heterogeneous robot policies
• capability boundaries
• distribution mismatch
• policy orchestration

Innovation

• memory-driven orchestration
• heterogeneous robot policies
• capability-aware routing
• policy handoff stabilization
• long-horizon planning