RGB-S: Image-Aligned Tactile Saliency for Robust Dexterous Manipulation

2026-06-07

AI Summary

This work addresses the challenge of robustly aligning tactile and visual modalities in scenarios where vision is unreliable or occluded. The authors propose an explicit image-domain alignment method that projects sparse tactile signals onto the RGB image plane using robot forward kinematics and camera calibration, generating force-modulated Gaussian saliency maps to model spatial uncertainty. They further introduce a zero-initialized conditional fusion architecture that injects physical contact priors into a pretrained visual backbone. This approach achieves the first explicit tactile-visual alignment directly in the image domain, leveraging geometric priors to enhance spatial reasoning. Evaluated on six real-world dexterous manipulation tasks under severe occlusion, the method improves success rates by 26.7 percentage points over the strongest implicit multimodal baseline, demonstrating significantly enhanced generalization and data efficiency.

Abstract

Effective visuo-tactile integration is critical for robotic dexterous manipulation, especially when visual observations are unreliable or occluded. However, robustly aligning sparse, heterogeneous tactile measurements with dense visual representations remains a fundamental challenge. Most existing approaches require policies to learn cross-modal correspondences implicitly from limited demonstrations, without leveraging geometric priors. As a result, they are often data-inefficient and generalize poorly when visual observations are degraded. To address this limitation, we propose a framework that explicitly grounds physical contacts in the image domain. Using robot forward kinematics and camera calibration, we project tactile sensor locations directly onto the RGB image plane. We then render force-modulated Gaussian saliency maps to model spatial uncertainty arising from kinematic and calibration errors. By integrating these 2D spatial anchors through a zero-initialized conditioning architecture, our method injects physical contact priors into standard visual backbones while preserving pre-trained visual representations. We evaluate our method on six dexterous manipulation tasks in both simulation and the real world under severe visual occlusions. Real-world experiments show that explicit RGB-S grounding in the image domain improves real-world occluded manipulation success rates by $26.7$ percentage points over the strongest implicit visuo-tactile baseline, suggesting its improved spatial reasoning and robustness to occlusion. Project page: touch-as-saliency.github.io

Problem

• visuo-tactile integration
• tactile-visual alignment
• dexterous manipulation
• occlusion robustness
• cross-modal correspondence

Innovation

• visuo-tactile integration
• tactile saliency
• image-aligned grounding
• dexterous manipulation
• spatial uncertainty modeling