TACT: Humanoid Whole-body Contact Manipulation through Deep Imitation Learning with Tactile Modality

2025-06-18

AI Summary

To address the high computational cost of motion generation and the difficulty of large-area tactile sensing in whole-body contact manipulation for humanoid robots, this paper proposes a deep imitation learning framework integrating high-resolution tactile modalities. Methodologically, it introduces, for the first time, deep integration of tactile signals into the Action Chunking with Transformers (ACT) architecture, jointly leveraging vision and proprioception; it further incorporates whole-body motion retargeting and real-time bipedal dynamic gait–balance control to enable stable contact manipulation during locomotion. Contributions include: (1) establishing the first imitation learning paradigm supporting high-density, whole-body tactile feedback; and (2) experimental validation on the full-scale RHP7 Kaleido humanoid platform, demonstrating significant improvements in robustness and task success rates for both large-area and fine-contact manipulation tasks.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Manipulation with whole-body contact by humanoid robots offers distinct advantages, including enhanced stability and reduced load. On the other hand, we need to address challenges such as the increased computational cost of motion generation and the difficulty of measuring broad-area contact. We therefore have developed a humanoid control system that allows a humanoid robot equipped with tactile sensors on its upper body to learn a policy for whole-body manipulation through imitation learning based on human teleoperation data. This policy, named tactile-modality extended ACT (TACT), has a feature to take multiple sensor modalities as input, including joint position, vision, and tactile measurements. Furthermore, by integrating this policy with retargeting and locomotion control based on a biped model, we demonstrate that the life-size humanoid robot RHP7 Kaleido is capable of achieving whole-body contact manipulation while maintaining balance and walking. Through detailed experimental verification, we show that inputting both vision and tactile modalities into the policy contributes to improving the robustness of manipulation involving broad and delicate contact.

Problem

• Develop humanoid control for whole-body contact manipulation
• Address challenges in motion generation and contact measurement
• Integrate vision and tactile sensors for robust manipulation

Innovation

• Deep imitation learning with tactile sensors
• Multi-sensor input including vision and touch
• Integration with retargeting and locomotion control