CAAT: Contact-Aware Attention Scaling and Tactile Masking for Data-Efficient Contact-Rich Manipulation

2026-08-02

AI Summary

This work addresses the limitation of existing Transformer-based visuo-tactile policies in contact-intensive manipulation tasks, which lack explicit contact-aware priors and struggle to efficiently model cross-modal representations. To overcome this, we propose the Contact-Aware Attention Transformer (CAAT) framework, which incorporates contact-aware attention scaling and dynamic tactile masking: it amplifies visual information prior to contact, emphasizes tactile signals upon contact, and suppresses interference from static background regions. Notably, CAAT requires no modifications to the action decoder and is readily compatible with diverse policy architectures. Experiments demonstrate that our method improves performance by 18.0 and 10.0 percentage points over naive fusion and gated fusion baselines, respectively, in simulation. On real-world robotic tasks, it achieves an average success rate of 60.0%, surpassing the strongest baseline by 21.1 percentage points.

Abstract

In contact-rich manipulation, visual observations primarily guide motion in free space, whereas tactile observations become particularly informative during contact. However, standard Transformer-based visuo-tactile policies typically rely on either token concatenation or learnable gating. These approaches lack explicit contact-aware priors, making it difficult to efficiently learn effective cross-modal representations from demonstrations. To address this limitation, we propose CAAT, a lightweight contact-aware framework that explicitly incorporates contact priors through attention scaling and dynamic tactile masking. Specifically, CAAT emphasizes visual information before contact and tactile information during contact. It also suppresses static background tokens by comparing the current tactile observation with a non-contact reference. CAAT can be integrated into commonly used Transformer-based policies without modifying their action decoders. In simulation, integrating CAAT with ACT improves the average success rate by 18.0 percentage points over direct visuo-tactile fusion and by 10.0 percentage points over gated fusion. In real-world experiments using a visuo-tactile UMI platform, CAAT achieves an average success rate of 60.0% across ACT, Diffusion Policy, and $π_0$, outperforming the strongest baseline by an average of 21.1 percentage points. These results demonstrate that explicit contact priors and dynamic tactile masking are effective in improving visuo-tactile policy learning and task performance of diverse policy architectures. https://mrjiangjm.github.io/caat/

Problem

• contact-rich manipulation
• visuo-tactile policies
• cross-modal representation
• data-efficient learning
• tactile sensing

Innovation

• contact-aware attention
• tactile masking
• visuo-tactile fusion
• data-efficient manipulation
• Transformer-based policy