FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation

2026-07-22

AI Summary

This work addresses the scarcity of tactile data, which hinders the application of vision–tactile fusion in robotic manipulation. The authors propose FELT, a novel framework that, for the first time, generates structured two-finger pressure tactile signals solely from RGB images. Leveraging a frozen large-scale vision encoder paired with a lightweight branch-query decoder, FELT predicts asymmetric contact patterns for left and right fingers in a single forward pass, and can output either tactile images or latent tactile features to augment purely visual policies. Notably, the method requires no real tactile data for training or deployment. Evaluated on four tasks with high contact complexity, FELT significantly outperforms vision-only baselines, with its latent-feature variant entirely eliminating reliance on physical tactile sensors.

Abstract

The sense of touch is central to manipulation, especially when vision is occluded or ambiguous. Although combining vision and touch improves manipulation, learning robust visuo-tactile policies requires substantial tactile data. Such data remains scarcer than visual data, because tactile sensors are fragile, specialized, and hard to standardize. To address this, we present Feature-Extracted Latent Tactile (FELT), a learning-based framework that synthesizes per-finger pressure tactile images from RGB observations, reducing the need for tactile-equipped data collection. FELT uses a large frozen visual encoder and a lightweight query decoder to predict tactile signals in a single feed-forward pass. To respect the physical topology of dual-finger tactile sensors, FELT decodes the left and right tactile sensor panels through separate branches, capturing the asymmetric contact patterns during interactions such as wiping, insertion, and in-hand rotation. At inference time, FELT only requires RGB data, allowing us to augment existing vision-only data with tactile observations, either as generated tactile images or as latent tactile features. Experiments on four contact-rich manipulation tasks demonstrate that both generated tactile images and latent tactile features improve policy success over vision-only baselines, with latent feature requiring no real tactile sensor during policy training or deployment. Supplementary material is available on our anonymous website: https://felt-tactile.github.io/.

Problem

• visuo-tactile manipulation
• tactile data scarcity
• tactile sensing
• robotic manipulation
• sensor standardization

Innovation

• visuo-tactile learning
• tactile signal synthesis
• latent tactile features
• manipulation policy
• sensor-free tactile augmentation