Learning with Object-centric Representations of Tactile Interactive Perception for Robot Manipulation

2026-09-27

AI Summary

This study addresses the challenge of inferring object physical properties and enabling adaptive robotic manipulation under sparse tactile signals. To this end, it proposes an object-centric, context-aware representation learning framework. Specifically, the method employs a Token Learner to autonomously select representative tactile segments and aligns tactile signals with descriptive text in a latent space via contrastive learning guided by semantic context, thereby achieving task-agnostic, generalizable object representations. Experimental results demonstrate that the proposed model attains property estimation accuracies of 93% and 84% on seen and unseen objects, respectively. Furthermore, multi-task manipulation success rates are significantly improved from 41% and 19% to 92% and 67%, highlighting the effectiveness of the learned representations for robust robotic interaction.

Abstract

Implicit object properties that are difficult to directly infer from vision, such as material, container contents, or softness, can be revealed through tactile sensing and exploratory interactions. However, because tactile signals are transient and sparse, extracting informative tactile events and effectively incorporating them into robotic manipulation remains a challenge. In this work, we present an object-centric context-aware manipulation framework that learns task-agnostic object representations through tactile exploration. A token learner autonomously selects representative tactile segments from long-horizon exploration, while contrastive alignment with descriptive text embeddings enables a latent space that captures multiple physical object properties. These learned representations are then used as semantic context to guide object-centric manipulation policies and adapt strategies based on object properties. Experiments show that the learned representations achieve 93% and 84% property estimation accuracy on seen and unseen objects. Evaluated on three tasks involving visually ambiguous objects, i.e. multi-object rearrangement, pouring, and box opening, the proposed framework improves both target selection and property-dependent manipulation adaptation, raising task success, aggregated over all evaluation trials, from 41% to 92% on seen objects and from 19% to 67% on unseen objects over baseline policies without object-context conditioning. Videos and additional results are available at https://xinyiyxyx.github.io/tactile-object-centric/.

Problem

• tactile perception
• robot manipulation
• object properties
• interactive perception

Innovation

• tactile interactive perception
• object-centric representation
• contrastive alignment
• token learner
• robot manipulation