Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations

2026-07-28

AI Summary

This work addresses the disconnect between high-level planning and low-level execution in long-horizon dexterous manipulation by proposing a novel framework that integrates Task and Motion Planning (TAMP) with imitation learning. Human demonstrations are decomposed into atomic skills based on contact relationships and modeled either as TAMP-compatible visuomotor policies or object-centric primitives, enabling, for the first time, seamless embedding of visuomotor strategies within TAMP. This approach reduces the data requirements for skill composition from exponential to linear in the number of skill types, substantially improving data efficiency and spatial generalization. Experiments demonstrate that the system achieves high success rates and strong robustness across diverse real-world and simulated scenarios, including unseen object configurations, multi-stage tasks, and complex physical constraints.

Abstract

Successfully automating dexterous, long-horizon robotic manipulation requires frameworks capable of both high-level reasoning and fine-grained execution. Traditional task and motion planning (TAMP), while excellent at symbolic planning, is often brittle in contact-rich operations. Simultaneously, imitation learning (IL), while effective in manipulation tasks with visual feedback, is limited by its low capability in spatial generalization and multi-stage operation. To reconcile their complementary strengths and limitations, we propose DR-LfD (Decomposed and Reorganized Skills Learned from Demonstrations), a framework that seamlessly integrates visuomotor policies into a TAMP-gated decision-making system. Based on contact relationships, DR-LfD decomposes human demonstrations into atomic skills, which are reproduced as visuomotor policies or object-centric primitives. The initiation, termination, and constraints of the visuomotor policies are carefully modeled and implemented in a TAMP-compatible form, enabling reorganization of skills learned from different sources. DR-LfD transforms the learning problem from one requiring exponential demonstration data over possible skill sequences to one whose demonstration burden scales with the number of distinct skill types, with limited data for each skill. Through comprehensive real-world and simulation benchmarking across diverse scenarios, we demonstrate the strong performance of DR-LfD on tasks involving multiple steps, unseen setups, and physical constraints. Project website: https://dr-lfd.github.io/DR-LfD-website.

Problem

• robotic manipulation
• task and motion planning
• imitation learning
• long-horizon tasks
• visuomotor policies

Innovation

• task and motion planning
• imitation learning
• visuomotor policies
• skill decomposition
• contact-based primitives