Factorized Tactile Representation and Control for Sim-to-Real Manipulation

2026-10-07

AI Summary

This study addresses the challenges of significant contact response discrepancies and control information loss in tactile sim-to-real transfer by proposing a decomposed tactile representation and control framework. The method introduces a novel factorized decomposition mechanism for tactile responses, decoupling contact dynamics into geometric, force distribution, and temporal variation components, which are independently evaluated through tailored encoding and randomization techniques. Additionally, a gating strategy is designed to preserve component-wise representations, enabling generalization under full-masking configurations without retraining. Experimental results demonstrate that the proposed framework achieves sub-millimeter contact localization, reduces force tracking error on unseen geometries to merely 1.69N, and improves success rates by 35% in real-world adversarial peg-in-hole tasks.

Abstract

Tactile sim-to-real learning must bridge simulated contact and device-specific sensor responses while preserving information needed for control. We propose a factorized tactile representation and control framework that maps normal force and contact patch to an effective contact response recoverable from sensor readings. The response is separated into contact geometry, force distribution, and temporal contact change, with representation-specific encoding and randomization. A Tactile Gated Policy preserves these representations separately through control and operates over all mask configurations without retraining. We evaluate the approach through response reconstruction, spatial alignment, force regulation, and contact-rich adversarial peg insertion in simulation and the real world, enabling the utility and transfer reliability of different tactile representations to be assessed independently. The approach achieves <1 mm contact localization, 1.69 N force-tracking error on unseen geometries, and a 35% improvement in real-world adversarial peg insertion over the unfactorized response, with different tactile representations benefiting different interactions.

Problem

• tactile sim-to-real
• manipulation
• contact representation
• force regulation
• peg insertion

Innovation

• Factorized Tactile Representation
• Sim-to-Real Transfer
• Tactile Gated Policy
• Contact-rich Manipulation
• Domain Randomization