Technology Category
Computer Vision: Multi-modal VisionIntelligent Robots: Multimodal Perception & Sensor FusionMachine Learning: Multimodal Learning
ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers
🤖 AI Summary
This work addresses three key limitations in vision–tactile multimodal fusion: overreliance on large foundation models, absence of multi-scale spatial modeling, and inadequate positional encoding design. To this end, we propose Vision–Tactile Positional Encodings (ViTaPEs)—a novel positional encoding scheme with provable injectivity, rigid-motion equivariance, and information preservation. ViTaPEs are the first to introduce multi-scale spatial reasoning into vision–tactile Transformers. Our method employs a lightweight Transformer architecture integrating cross-modal attention and equivariant modeling, eliminating the need for pre-trained vision-language models while enabling robust cross-modal alignment and task-agnostic representation learning. Extensive experiments demonstrate state-of-the-art performance across multiple real-world datasets, support zero-shot cross-domain generalization, and significantly improve success-rate prediction accuracy in robotic grasping tasks.
Technology Category
Application Category
📝 Abstract
Tactile sensing provides local essential information that is complementary to visual perception, such as texture, compliance, and force. Despite recent advances in visuotactile representation learning, challenges remain in fusing these modalities and generalizing across tasks and environments without heavy reliance on pre-trained vision-language models. Moreover, existing methods do not study positional encodings, thereby overlooking the multi-scale spatial reasoning needed to capture fine-grained visuotactile correlations. We introduce ViTaPEs, a transformer-based framework that robustly integrates visual and tactile input data to learn task-agnostic representations for visuotactile perception. Our approach exploits a novel multi-scale positional encoding scheme to capture intra-modal structures, while simultaneously modeling cross-modal cues. Unlike prior work, we provide provable guarantees in visuotactile fusion, showing that our encodings are injective, rigid-motion-equivariant, and information-preserving, validating these properties empirically. Experiments on multiple large-scale real-world datasets show that ViTaPEs not only surpasses state-of-the-art baselines across various recognition tasks but also demonstrates zero-shot generalization to unseen, out-of-domain scenarios. We further demonstrate the transfer-learning strength of ViTaPEs in a robotic grasping task, where it outperforms state-of-the-art baselines in predicting grasp success. Project page: https://sites.google.com/view/vitapes
Problem
Research questions and friction points this paper is trying to address.
Fusing visual and tactile modalities without pre-trained models
Capturing multi-scale spatial visuotactile correlations
Achieving zero-shot generalization in cross-modal tasks
Innovation
Methods, ideas, or system contributions that make the work stand out.
Transformer-based fusion of visual and tactile data
Multi-scale positional encoding for spatial reasoning
Injective, equivariant, information-preserving visuotactile encodings
🔎 Similar Papers
No similar papers found.
