VTLoc: Learning-based Tactile Contact Localization in Visual Point Clouds

2026-07-17

AI Summary

This work addresses the challenges of spatial misalignment between visual and tactile modalities and the ambiguity in mapping local tactile signals to global visual geometry for contact point localization. To overcome these issues, the authors propose VTLoc, a novel framework that takes 3D point clouds as visual input and integrates a geometric multimodal alignment module with an iterative localization updater to effectively fuse visual-tactile features and achieve cross-modal spatial alignment. A pseudo point cloud reconstruction mechanism is further introduced to enable joint optimization. As the first learning-based method to perform visual-tactile contact point localization on real-world objects, VTLoc significantly improves single-point localization accuracy on a new benchmark comprising 100 real objects, effectively resolving the local-to-global correspondence ambiguity.

Abstract

Vision and touch are complementary modalities essential for robotic perception and manipulation. While vision provides global object context, touch offers precise local information at contact points. Integrating these modalities for contact localization, i.e., predicting the location of touch on an object's surface, poses significant challenges due to the need for accurate spatial alignment between tactile data and visual geometry. To address this challenge, we propose VTLoc, a novel visual-tactile framework that localizes contact points from tactile readings using a 3D point cloud as visual input. VTLoc introduces two key components: a geometric multi-modal alignment module, which reconstructs a pseudo-point cloud from fused visual-tactile features and aligns it with the visual point cloud to enforce spatial consistencies across modalities; and an iterative localizing updater, which iteratively refines the predicted contact location using fused visual-tactile features. Evaluated on a new benchmark of 100 real-world objects, VTLoc improves single-touch contact localization by reducing local-to-global correspondence ambiguity.

Problem

• contact localization
• visual-tactile integration
• spatial alignment
• point cloud
• multimodal perception

Innovation

• visual-tactile fusion
• contact localization
• 3D point cloud
• multi-modal alignment
• iterative refinement