TouchTherm: Building Multimodal Digital Twins of Objects for Tactile and Thermal Rendering

2026-10-01

AI Summary

This study addresses the inability of existing methods to model micro-tactile and transient thermal dynamics, which hinders high-fidelity multimodal interaction. We propose the first vision-tactile-thermal digital twin framework integrating micro-surface reconstruction with temperature dynamics. Specifically, structured light and photometric stereo are combined to recover micro-height fields, while dynamic thermal fields are reconstructed from multi-view infrared videos via physics-based regularization, enabling simulation-ready asset generation. Experimental results demonstrate a thermal field mean absolute error as low as 0.465°C, effectively supporting tactile recognition and spatiotemporal thermal feedback demonstrations using VR gloves. By bridging the data gap in tactile and thermal rendering, this work provides a comprehensive foundation for realistic multisensory simulation and interaction.

Abstract

Robotic simulation and virtual reality increasingly require object assets that capture not only visual geometry but also the physical cues underlying tactile and thermal interaction. Existing 3D datasets and reconstruction methods primarily represent object-scale geometry and visual appearance, overlooking microscale surface structure for high-fidelity haptic rendering and transient temperature dynamics for temperature-aware interaction. We present TouchTherm, a framework for constructing simulation-ready visuo-tactile-thermal object assets from real-world objects. For visual and tactile reconstruction, we combine structured-light scanning with multiview normal maps obtained from photometric stereo. The normal maps are registered to the scanned geometry and transformed into tangent space to recover local micro-height fields for optical tactile rendering, while the coarse mesh handles collision detection. For thermal reconstruction, we capture synchronized multiview infrared videos of natural cooling following controlled heating and reconstruct a physics-regularized dynamic thermal field. Experiments on 20 objects show that the reconstructed micro-height fields preserve dominant surface structures and recover higher-frequency details beyond the coarse geometry, while the thermal fields achieve held-out surface-temperature MAEs of 0.465 degrees C and 0.592 degrees C at 30 s and 45 s, respectively. The resulting tactile assets support synthetic-to-real object recognition from tactile observations, while a glove-based VR system demonstrates spatially and temporally varying thermal feedback. These results highlight the potential of TouchTherm for multimodal sensory simulation and temperature-aware virtual interaction.

Problem

• multimodal digital twins
• tactile rendering
• thermal interaction
• microscale surface structure
• 3D reconstruction

Innovation

• Multimodal Digital Twins
• Tactile Rendering
• Thermal Reconstruction
• Photometric Stereo
• Physics-regularized Dynamic Thermal Field