DEXTERA: From a Single Image to Deployable Dexterous Manipulation via Real-to-Sim-to-Real

2026-09-17

AI Summary

本文提出DEXTERA框架，通过单张图像生成可部署的灵巧操作策略，解决了从仿真到现实迁移中的视觉、几何和动力学差距问题。

Abstract

Collecting real-world robot data for dexterous manipulation is costly and time-consuming. While high-fidelity physics simulators enable scalable data synthesis and policy learning, constructing deployment-ready digital twins manually remains labor-intensive, and residual visual, geometric, and dynamics gaps hinder reliable sim-to-real transfer. We present DEXTERA, an automated real-to-sim-to-real framework that transforms a single RGB image into deployable policies for dexterous manipulation across four unified stages: (1) single-image scene factorization into a static Gaussian background and interactive rigid or articulated assets with VLM-inferred physical parameters; (2) metric scene global alignment, object canonicalization, and morphology-balanced robot calibration; (3) scalable simulator task primitive construction, VR teleoperation, and object-centric trajectory synthesis; and (4) a shared multimodal policy interface supporting both imitation learning and reinforcement learning. We evaluate DEXTERA across 13 task-embodiment pairs, 2 dexterous robot platforms, and 6 policy architectures. Experimental results demonstrate that DEXTERA achieves superior visual fidelity and 3D geometric reconstruction compared to generative baselines, while cross-domain trajectory replays validate strong physical interaction consistency. Furthermore, simulation-only trained policies enable viable zero-shot real-robot deployment, while simulation-real co-training substantially improves mean physical policy success from 29.2% to 61.9% across diverse policy architectures.

Problem

• Dexterous Manipulation
• Sim-to-Real Transfer
• Digital Twins
• Physics Simulators
• Single Image

Innovation

• Real-to-Sim-to-Real
• Single-Image Scene Factorization
• Automated Framework
• Dexterous Manipulation
• Zero-Shot Real-Robot Deployment