kinesthetic teaching

Significant visual and kinematic discrepancies between human hand demonstrations and robotic manipulation, coupled with reliance on specialized teleoperation hardware, hinder scalable and cost-effective demonstration data collection. Method: We propose an end-to-end gesture-to-gripper motion generation framework. Using a wrist-mounted GoPro fisheye camera, we capture first-person gesture videos and construct a paired human-hand–robot SE(3) action dataset. A spatiotemporally aligned generative model directly maps hand keypoint sequences to robot gripper trajectories—without requiring physical robots during demonstration recording. Contribution/Results: This work introduces the first calibration-free, teleoperation-free cross-modal motion generation method based solely on monocular fisheye video. Experiments show that policies trained on generated demonstrations achieve performance comparable to those trained on ground-truth demonstrations across diverse dexterous manipulation tasks. Data acquisition efficiency improves by over 5×, substantially enhancing the practicality and scalability of robotic imitation learning.

Constructing high-quality, cross-hardware datasets for contact-rich robotic manipulation remains challenging due to high data acquisition costs and hardware-specific constraints. Method: This paper proposes a low-cost, high-fidelity multi-source data generation framework. It introduces the first “embodiment-flexible” VR-based human demonstration capture system, integrated with physics simulation, kinematic retargeting, and constraint-aware, multi-body parameter-adaptive trajectory optimization—enabling cross-configuration data reuse and zero-shot transfer. Contribution/Results: Using this pipeline, we build a large-scale contact manipulation dataset and train a diffusion-based policy with strong generalization. Evaluated on Allegro Hand and dual-arm iiwa platforms, the policy achieves zero-shot deployment on real dual-arm iiwa hardware, significantly improving success rates on contact-intensive tasks with minimal human intervention. Key contributions are the embodiment-flexible data collection paradigm and the physically consistent, parameter-adaptive optimization mechanism.

This work addresses two core challenges in large-scale robotic manipulation datasets: (1) designing high-value diversity dimensions to enhance data utility, and (2) efficiently retrieving task-aligned demonstrations from existing datasets. To this end, we introduce a programmable data generation framework that explicitly models controllable diversity variables—including camera pose, object categories, and spatial layout. Our analysis reveals, for the first time, that camera pose and spatial arrangement are critical determinants of both dataset diversity and task alignment. We further propose a task-oriented demonstration retrieval algorithm grounded in geometric-semantic joint alignment. Evaluated on real-world datasets including DROID, our method improves downstream policy performance by up to 70%. Crucially, insights and gains observed in simulation generalize successfully to physical robot platforms, demonstrating robust cross-domain transferability.

Demonstrating high-quality, physically plausible trajectories for teleoperated dexterous manipulation in contact-rich environments remains challenging due to the difficulty of acquiring consistent, diverse, and kinematically feasible human demonstrations. Method: This paper proposes a model-driven trajectory generation framework. It first identifies the high-entropy, low-consistency behavior of sampling-based planners (e.g., RRT) in contact-rich settings; then introduces a three-stage pipeline—RRT initialization, MPC-based refinement, and diffusion-model-based resampling—to jointly ensure physical feasibility, consistency, and diversity. Furthermore, it develops a goal-conditioned diffusion behavioral cloning (DBC) policy. Results: The method achieves zero-shot hardware transfer on two challenging contact-rich manipulation tasks, outperforming conventional behavioral cloning and pure planning baselines in terms of success rate, robustness, and generalization—without requiring any real-world demonstration data.

Existing robot manipulation policies suffer from limited generalization due to reliance on small-scale, low-diversity simulation data or single-environment real-world datasets. To address this, we introduce DROID—the first large-scale, cross-household, real-world distributed robot manipulation dataset. It encompasses 564 diverse household environments, 84 task categories, and 76k high-quality trajectories (350 hours), collected over 12 months by 50 geographically distributed contributors. DROID pioneers intercontinental, multi-brand robotic hardware coordination (UR5e and Franka Emika arms) via remote distributed data collection, integrating standardized interfaces, precise action alignment, and rigorous quality filtering. We fully open-source the hardware specifications, data collection infrastructure, and training code. Policies trained on DROID achieve a 27% average success rate improvement in cross-scene generalization benchmarks and demonstrate superior zero-shot transfer performance compared to prior state-of-the-art methods.

Training general-purpose bimanual dexterous manipulation policies is hindered by the scarcity of high-quality, diverse data, as existing synthetic approaches struggle to simultaneously ensure task diversity and robotic feasibility. This work proposes the first reinforcement learning data generation framework that integrates a universal reward mechanism, domain randomization, and language-conditioned annotations to systematically produce diverse yet executable bimanual manipulation data. The method substantially enhances the generalization performance of language-guided multitask policies across three representative dexterous manipulation tasks, effectively overcoming the longstanding trade-off between diversity and feasibility that plagues conventional data generation paradigms.

This work addresses the poor policy generalization and low data efficiency in robotic manipulation caused by the visual discrepancies between simulation and reality (sim-to-real) and the embodiment mismatch between humans and robots (human-to-robot). To bridge these dual gaps, we propose SimHum, a co-training framework that systematically integrates kinematic priors from simulated robot trajectories with visual priors derived from real human demonstrations. Through joint training, SimHum effectively aligns these complementary sources of information. Under identical data budgets, SimHum achieves up to a 40% performance improvement over baselines. Notably, with only 80 real human demonstrations, it attains a 62.5% success rate on out-of-domain tasks—surpassing purely real-data baselines by a factor of 7.1.

This work addresses the lack of systematic evaluation for Vision-Language-Action (VLA) models in sim-to-real transfer for dexterous manipulation tasks by introducing a real-world evaluation protocol that encompasses variations in background, lighting, distractors, object types, and spatial configurations. Through over 10,000 physical trials, the study systematically investigates the impact of multi-level domain randomization, photorealistic rendering, high-fidelity physics modeling, and reinforcement learning strategies on generalization performance. It presents the first comprehensive assessment of VLA models’ sim-to-real capabilities in real-world dexterous manipulation, establishes a standardized benchmark, and releases an open-source robotic platform, thereby significantly enhancing the robustness and reproducibility of policies in complex real-world scenarios.

This work addresses the challenge of collecting feasible demonstration trajectories during handheld gripper data acquisition without robot hardware involvement, where real-time assessment of trajectory executability on the target robot is typically infeasible, leading to abundant invalid demonstrations. To this end, the authors propose FeasibleCap, a system that— for the first time—integrates real-time reachability analysis, joint velocity limits, and collision checking based on the target robot’s kinematic model into a lightweight handheld capture paradigm. By leveraging on-device visual overlays and haptic feedback, FeasibleCap guides users to instantly correct infeasible motions, enabling closed-loop guidance without reliance on learned models or head-mounted displays. Experiments demonstrate that the approach significantly improves replay success rates and reduces the proportion of infeasible frames in pick-and-place and throwing tasks, with particularly pronounced gains in the latter, while maintaining strong cross-platform transferability.

This work addresses the lack of scalable, trustworthy evaluation methods and physically plausible training data for general-purpose robotic policies, compounded by the high cost and poor reproducibility of real-world robot experiments. To overcome these challenges, the authors propose a human↔simulation↔robot bidirectional alignment framework supported by a cloud-native toolchain. Leveraging the JoySim simulator—integrated with reconstruction, rendering, and realism-enhancement modules—they implement a high-fidelity digital twin on the JD Cloud platform. Human demonstrations are transformed into physically consistent trajectories, annotations, and visual observations, while simulation serves dual roles as a scalable evaluation layer and a data filter. This approach substantially improves both the efficiency of data generation and the reliability of policy evaluation.