Robust Bayesian Scene Reconstruction with Retrieval-Augmented Priors for Precise Grasping and Planning

2024-11-29

AI Summary

Reconstructing multiple objects from a single RGB-D frame faces core challenges including geometric unobservability in occluded regions, poor noise robustness, weak generalization, and miscalibrated confidence estimation. Method: We propose a retrieval-augmented Bayesian reconstruction framework. It introduces, for the first time, a cross-modal part-level retrieval mechanism that dynamically matches semantically similar parts from a pre-trained 3D mesh library to construct an information-rich, adaptive Bayesian prior—jointly modeling geometric uncertainty and occlusion reasoning. The framework integrates RGB-D perception, differentiable rendering, and probabilistic inference to yield well-calibrated reconstruction confidence. Contribution/Results: Evaluated on both synthetic and real-world cluttered scenes, our method significantly improves reconstruction accuracy in occluded regions and enhances robustness to sensor noise. Crucially, it enables dexterous robotic hands to achieve high-success-rate grasping, demonstrating practical applicability in complex, unstructured environments.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Biometrics, Face, Gesture & Pose
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Constructing 3D representations of object geometry is critical for many robotics tasks, particularly manipulation problems. These representations must be built from potentially noisy partial observations. In this work, we focus on the problem of reconstructing a multi-object scene from a single RGBD image using a fixed camera. Traditional scene representation methods generally cannot infer the geometry of unobserved regions of the objects in the image. Attempts have been made to leverage deep learning to train on a dataset of known objects and representations, and then generalize to new observations. However, this can be brittle to noisy real-world observations and objects not contained in the dataset, and do not provide well-calibrated reconstruction confidences. We propose BRRP, a reconstruction method that leverages preexisting mesh datasets to build an informative prior during robust probabilistic reconstruction. We introduce the concept of a retrieval-augmented prior, where we retrieve relevant components of our prior distribution from a database of objects during inference. The resulting prior enables estimation of the geometry of occluded portions of the in-scene objects. Our method produces a distribution over object shape that can be used for reconstruction and measuring uncertainty. We evaluate our method in both simulated scenes and in the real world. We demonstrate the robustness of our method against deep learning-only approaches while being more accurate than a method without an informative prior. Through real-world experiments, we particularly highlight the capability of BRRP to enable successful dexterous manipulation in clutter.

Problem

• Reconstructing multi-object scenes from single RGBD images with occlusions
• Addressing brittleness of deep learning methods to noisy observations
• Providing calibrated uncertainty estimates for robotic grasping tasks

Innovation

• Retrieval-augmented prior from mesh databases
• Probabilistic reconstruction with uncertainty measurement
• Robust geometry estimation for occluded object portions