Bayesian Active Object Recognition and 6D Pose Estimation from Multimodal Contact Sensing

2026-03-22

AI Summary

This work addresses the challenge of inaccurate object recognition and six-degree-of-freedom pose estimation in tactile interaction, which arises from perceptual ambiguity and contact uncertainty. To this end, the authors propose a Bayesian active tactile exploration framework that unifies multimodal tactile signals—specifically GelSight images and wrist force/torque measurements—with non-contact free-space constraints within a joint belief update mechanism. The framework employs a tailored particle filter and an information-maximizing exploration strategy subject to reachability constraints to dynamically optimize subsequent touch actions. Experiments on a Franka Panda platform with 11 YCB objects demonstrate that the proposed approach significantly improves accuracy and robustness in both recognition and pose estimation compared to baseline methods relying solely on force/torque sensing, while also reducing the number of required interactions.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Activity and Plan Recognition
• Computer Vision: Biometrics, Face, Gesture & Pose

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We present an active tactile exploration framework for joint object recognition and 6D pose estimation. The proposed method integrates wrist force/torque sensing, GelSight tactile sensing, and free-space constraints within a Bayesian inference framework that maintains a belief over object class and pose during active tactile exploration. By combining contact and non-contact evidence, the framework reduces ambiguity and improves robustness in the joint class-pose estimation problem. To enable efficient inference in the large hypothesis space, we employ a customized particle filter that progressively samples particles based on new observations. The inferred belief is further used to guide active exploration by selecting informative next touches under reachability constraints. For effective data collection, a motion planning and control framework is developed to plan and execute feasible paths for tactile exploration, handle unexpected contacts and GelSight-surface alignment with tactile servoing. We evaluate the framework in simulation and on a Franka Panda robot using 11 YCB objects. Results show that incorporating tactile and free-space information substantially improves recognition and pose estimation accuracy and stability, while reducing the number of action cycles compared with force/torque-only baselines. Code, dataset, and supplementary material will be made available online.

Problem

• object recognition
• 6D pose estimation
• tactile sensing
• active exploration
• Bayesian inference

Innovation

• Bayesian inference
• active tactile exploration
• multimodal contact sensing
• 6D pose estimation
• particle filter