MoDex: Planning High-Dimensional Dexterous Control via Learning Neural Internal Models

2024-09-17

AI Summary

To address challenges in high-dimensional dexterous hand control—including low data efficiency, poor task generalization, and difficult sim-to-real transfer—this paper draws inspiration from the human internal model mechanism and introduces, for the first time, a neural internal model framework for dexterous manipulation. Our approach proposes a learnable hand neuromechanical dynamics model integrated with a bidirectional optimal control planning architecture. It synergistically incorporates external vision models and large language models (LLMs) to enable both dexterous manipulation planning and multi-pose gesture generation driven by natural-language instructions. Evaluated across multiple dexterous hands, the framework achieves substantial improvements: over 50% reduction in training sample requirements; efficient cross-task and cross-hardware transfer; robust and generalizable dexterous manipulation in both simulation and real-world settings; and LLM-guided, semantically grounded hand pose synthesis.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

Controlling hands in high-dimensional action space has been a longstanding challenge, yet humans naturally perform dexterous tasks with ease. In this paper, we draw inspiration from the concept of internal model exhibited in human behavior and reconsider dexterous hands as learnable systems. Specifically, we introduce MoDex, a framework that includes a couple of neural networks (NNs) capturing the dynamical characteristics of hands and a bidirectional planning approach, which demonstrates both training and planning efficiency. To show the versatility of MoDex, we further integrate it with an external model to manipulate in-hand objects and a large language model (LLM) to generate various gestures in both simulation and real world. Extensive experiments on different dexterous hands address the data efficiency in learning a new task and the transferability between different tasks.

Problem

• Planning high-dimensional dexterous hand control efficiently
• Learning neural internal models for hand dynamics
• Integrating models for object manipulation and gesture generation

Innovation

• Neural networks model hand dynamics
• Bidirectional planning enhances efficiency
• Integrates LLM for gesture generation