Language-Guided Object-Centric Diffusion Policy for Generalizable and Collision-Aware Robotic Manipulation

2024-06-29

AI Summary

To address poor generalization and lack of collision awareness in robotic manipulation under cluttered scenes, viewpoint variations, and interference from visually similar objects, this paper proposes a language-guided object-centric diffusion policy. The method end-to-end integrates large language model (LLM)-based semantic parsing, open-vocabulary 3D instance segmentation, object-centric point cloud representation, and cost-guided denoising—achieving zero-shot, training-free collision avoidance and cross-scene adaptive manipulation for the first time. Key contributions are: (1) a time-invariant cost-guided generative mechanism based on clean trajectory estimation; and (2) a synergistic modeling framework unifying LLM, 3D segmentation, and diffusion policy. Evaluated on 21 RLBench tasks, the approach achieves 68.7% success rate using only 40 demonstrations—outperforming state-of-the-art 2D and 3D baselines by 29% and 25%, respectively—and maintains robust generalization and collision-free performance under unseen conditions including occlusion and viewpoint shifts.

Technology Category

• Intelligent Robots: Manipulation
• Natural Language Processing: Safety and Robustness
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Learning from demonstrations faces challenges in generalizing beyond the training data and often lacks collision awareness. This paper introduces Lan-o3dp, a language-guided object-centric diffusion policy framework that can adapt to unseen situations such as cluttered scenes, shifting camera views, and ambiguous similar objects while offering training-free collision avoidance and achieving a high success rate with few demonstrations. We train a diffusion model conditioned on 3D point clouds of task-relevant objects to predict the robot's end-effector trajectories, enabling it to complete the tasks. During inference, we incorporate cost optimization into denoising steps to guide the generated trajectory to be collision-free. We leverage open-set segmentation to obtain the 3D point clouds of related objects. We use a large language model to identify the target objects and possible obstacles by interpreting the user's natural language instructions. To effectively guide the conditional diffusion model using a time-independent cost function, we proposed a novel guided generation mechanism based on the estimated clean trajectories. In the simulation, we showed that diffusion policy based on the object-centric 3D representation achieves a much higher success rate (68.7%) compared to baselines with simple 2D (39.3%) and 3D scene (43.6%) representations across 21 challenging RLBench tasks with only 40 demonstrations. In real-world experiments, we extensively evaluated the generalization in various unseen situations and validated the effectiveness of the proposed zero-shot cost-guided collision avoidance.

Problem

• Generalizing robotic manipulation beyond training data
• Achieving collision-free trajectories in cluttered scenes
• Interpreting natural language for object and obstacle identification

Innovation

• Language-guided diffusion policy for robotic manipulation
• Training-free collision avoidance with cost optimization
• Open-set segmentation for 3D object point clouds