Omni-Scan: Creating Visually-Accurate Digital Twin Object Models Using a Bimanual Robot with Handover and Gaussian Splat Merging

2025-08-01

AI Summary

Traditional 3D scanning methods—such as multi-camera arrays, laser scanning, or wrist-mounted cameras—are constrained by limited workspace and struggle to achieve omnidirectional object modeling. To address this, we propose a dual-arm collaborative vision-scanning framework: two robotic arms jointly perform object grasping, handover, and multi-pose rotation, while a fixed monocular camera captures full-coverage imagery. Robust foreground segmentation and background suppression are achieved via DepthAnything, SegmentAnything, and RAFT optical flow models. We introduce a novel handover mechanism to mitigate gripper occlusion and enhance the 3D Gaussian Splatting (3DGS) training pipeline to support multi-stage view registration and point-cloud fusion. The resulting high-fidelity digital twin enables fully automated, omnidirectional 3D reconstruction. Evaluated on defect detection across 12 industrial and household parts, our method achieves an average accuracy of 83% and supports interactive 3D visualization.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

3D Gaussian Splats (3DGSs) are 3D object models derived from multi-view images. Such "digital twins" are useful for simulations, virtual reality, marketing, robot policy fine-tuning, and part inspection. 3D object scanning usually requires multi-camera arrays, precise laser scanners, or robot wrist-mounted cameras, which have restricted workspaces. We propose Omni-Scan, a pipeline for producing high-quality 3D Gaussian Splat models using a bi-manual robot that grasps an object with one gripper and rotates the object with respect to a stationary camera. The object is then re-grasped by a second gripper to expose surfaces that were occluded by the first gripper. We present the Omni-Scan robot pipeline using DepthAny-thing, Segment Anything, as well as RAFT optical flow models to identify and isolate objects held by a robot gripper while removing the gripper and the background. We then modify the 3DGS training pipeline to support concatenated datasets with gripper occlusion, producing an omni-directional (360 degree view) model of the object. We apply Omni-Scan to part defect inspection, finding that it can identify visual or geometric defects in 12 different industrial and household objects with an average accuracy of 83%. Interactive videos of Omni-Scan 3DGS models can be found at https://berkeleyautomation.github.io/omni-scan/

Problem

• Creating high-quality 3D object models using bimanual robot
• Overcoming occlusion issues in 3D scanning with gripper handover
• Enabling omni-directional object inspection for defect detection

Innovation

• Bimanual robot with handover for 360-degree scanning
• DepthAnything and Segment Anything for object isolation
• Modified 3DGS training for gripper occlusion handling