LEGATO: Cross-Embodiment Imitation Using a Grasping Tool

2024-11-06

arXiv.org

AI Summary

To address low efficiency and poor generalization in cross-morphology robotic skill transfer, this paper proposes LEGATO—a framework that unifies action and observation spaces across heterogeneous robot morphologies via a handheld gripper, enabling embodiment-agnostic task definition. Methodologically, LEGATO decouples task semantics from body-specific kinodynamic constraints during visuomotor policy training by jointly leveraging motion-invariance loss functions and inverse-kinematics (IK)-based motion retargeting. Crucially, it achieves zero-shot skill transfer across diverse robot platforms—such as manipulators, quadrupeds, and bimanual systems—without any fine-tuning on the target robot. Experiments in both simulation and real-world settings demonstrate that LEGATO significantly improves skill reusability and deployment robustness. By abstracting embodiment into a standardized, portable interface, LEGATO establishes a scalable, embodiment-agnostic paradigm for general-purpose robotic learning.

Technology Category

• Intelligent Robots: Embodied AI
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Cross-embodiment imitation learning enables policies trained on specific embodiments to transfer across different robots, unlocking the potential for large-scale imitation learning that is both cost-effective and highly reusable. This paper presents LEGATO, a cross-embodiment imitation learning framework for visuomotor skill transfer across varied kinematic morphologies. We introduce a handheld gripper that unifies action and observation spaces, allowing tasks to be defined consistently across robots. We train visuomotor policies on task demonstrations using this gripper through imitation learning, applying transformation to a motion-invariant space for computing the training loss. Gripper motions generated by the policies are retargeted into high-degree-of-freedom whole-body motions using inverse kinematics for deployment across diverse embodiments. Our evaluations in simulation and real-robot experiments highlight the framework's effectiveness in learning and transferring visuomotor skills across various robots. More information can be found at the project page: https://ut-hcrl.github.io/LEGATO.

Problem

• Robot Learning
• Task Generalization
• Morphological Adaptation

Innovation

• Cross-morphology Imitation Learning
• Tool-mediated Skill Transfer
• Robot Generalization