FeelWorld: Visuo-Tactile World Model for Hierarchical Contact Prediction and Planning

2026-07-27

AI Summary

This work addresses the limitation of existing visual world models in accurately predicting contact-rich physical interactions due to their neglect of tactile information. To overcome this, the authors propose a hierarchical visuo-tactile world model that jointly predicts future visual and tactile latent variables—including contact state, 3D forces, and slip—through a novel contact-gated asymmetric attention mechanism that dynamically switches between distinct dynamics pathways before and after contact. This approach is the first to incorporate explicit tactile supervision into a world model, integrating shared latent dynamics, autoregressive rollout prediction, contextual noise injection, and a contact-aware CEM planning algorithm. The method achieves a 10-step LPIPS of 0.058 across multiple tasks, reduces 80-step rollout prediction error by 61% compared to purely visual baselines, and attains an 81.7% average success rate in zero-shot planning.

Abstract

Humans plan physical interactions by imagining the possible outcomes of candidate actions. However, existing visual world models primarily capture appearance dynamics while overlooking the tactile states that govern contact-rich interactions, potentially producing imagined futures that appear visually plausible but violate physical dynamics. We introduce FeelWorld, a hierarchical visuo-tactile world model that jointly predicts future visual latents and three tactile states. FeelWorld organizes these states hierarchically as contact state, a 3D tactile latent that encodes force-related information, and slip state. These states are jointly predicted by a shared latent dynamics model with explicit supervision. To prevent irrelevant tactile signals during free-space motion from degrading visual prediction, we introduce a contact-gated asymmetric attention mechanism that maintains a visual-only prediction pathway before contact and enables joint visuo-tactile dynamics prediction during contact. The model is further trained with autoregressive rollouts and context noise injection to improve robustness to compounding errors. The predicted contact and slip states also support contact-aware CEM planning. Experiments on chip grasping, fruit grasping, and USB insertion show that FeelWorld reduces 10-step LPIPS from 0.084 to 0.058 and maintains an LPIPS that is 61% lower than that of the visual baseline after an 80-step autoregressive rollout. FeelWorld also achieves an average zero-shot planning success rate of 81.7%, providing an effective approach for incorporating tactile sensing into world models.

Problem

• visuo-tactile world model
• contact prediction
• tactile states
• physical dynamics
• hierarchical planning

Innovation

• visuo-tactile world model
• hierarchical contact prediction
• contact-gated attention
• tactile state estimation
• contact-aware planning