HapticWorld: an Interactive World Simulator with Real-time Torque Feedback

2026-09-25

AI Summary

This study addresses the scarcity of force sensing data in contact-rich manipulation and the absence of tactile feedback in visual world models. We propose a scalable, interactive world model that integrates real-time torque feedback, pioneering the incorporation of joint torque prediction and tactile closed-loop rendering into visual world models to enable joint visual-tactile generation and human-robot interactive feedback. Experimental results demonstrate that the proposed framework improves data collection efficiency by 1.6× and achieves a real-world policy success rate of 90% (54/60). This performance approaches the theoretical upper bound and significantly outperforms purely visual baselines, highlighting the effectiveness of integrating proprioceptive and tactile modalities within predictive world models for robotic manipulation.

Abstract

Contact-rich manipulation depends on force sensing that is hard to infer from visual signals alone, both for collecting demonstrations and for training policies. Force-annotated data, however, remains hard to obtain at scale: real-robot collection ties every demonstration to physical hardware, physics simulators report contact forces that deviate systematically from real measurements, and learned world simulators, though scalable and realistic, are vision-only, so operators feel nothing during data collection and the data carries no force/torque (F/T) labels. We present HapticWorld, an interactive world simulator that predicts joint torque together with observations and renders it back to the operator in real time, closing the haptic loop between a human and a learned world model. Across three contact-rich tasks, torque feedback raises data collection throughput by 1.6 times on average. Policies trained on HapticWorld-generated demonstrations succeed in 54/60 real-world trials, approaching the 56/60 upper bound of real-world data, and far exceeding the 19/60 success rate of the vision-only baseline. Moreover, the success rates measured inside HapticWorld closely match real-world evaluation, demonstrating that HapticWorld can serve as a stand-alone F/T-conditioned policy evaluation platform.

Problem

• contact-rich manipulation
• force/torque feedback
• world simulator
• haptic data collection
• robot learning

Innovation

• Interactive World Simulator
• Real-time Torque Feedback
• Contact-rich Manipulation
• Haptic Loop
• Policy Evaluation