FuseGrasp: Radar-Camera Fusion for Robotic Grasping of Transparent Objects

2025-02-27

AI Summary

Transparent objects impede visual recognition and depth estimation due to optical transparency, severely degrading robotic grasping performance—especially under challenging conditions such as low illumination. To address this, we propose FuseGrasp, the first millimeter-wave (mmWave) radar–camera fusion system designed specifically for manipulation of transparent objects. Our method introduces a dual-modality deep neural network that jointly performs depth completion and material classification; leverages mmWave radar’s non-penetrative interaction with transparent materials (e.g., glass, plastic) to enhance perception; and employs a two-stage transfer learning strategy to mitigate scarcity of labeled radar data. Experiments demonstrate significant improvements in depth reconstruction accuracy and material classification precision. Deployed on a real robotic platform, FuseGrasp achieves high grasping success rates and maintains robustness under adverse conditions—including low light, specular reflections, and occlusions.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Transparent, Interpretable, Explainable ML

Application Category

• Responsible Web: Algorithmic accountability and transparency on the web
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Data transparency and provenance

Abstract

Transparent objects are prevalent in everyday environments, but their distinct physical properties pose significant challenges for camera-guided robotic arms. Current research is mainly dependent on camera-only approaches, which often falter in suboptimal conditions, such as low-light environments. In response to this challenge, we present FuseGrasp, the first radar-camera fusion system tailored to enhance the transparent objects manipulation. FuseGrasp exploits the weak penetrating property of millimeter-wave (mmWave) signals, which causes transparent materials to appear opaque, and combines it with the precise motion control of a robotic arm to acquire high-quality mmWave radar images of transparent objects. The system employs a carefully designed deep neural network to fuse radar and camera imagery, thereby improving depth completion and elevating the success rate of object grasping. Nevertheless, training FuseGrasp effectively is non-trivial, due to limited radar image datasets for transparent objects. We address this issue utilizing large RGB-D dataset, and propose an effective two-stage training approach: we first pre-train FuseGrasp on a large public RGB-D dataset of transparent objects, then fine-tune it on a self-built small RGB-D-Radar dataset. Furthermore, as a byproduct, FuseGrasp can determine the composition of transparent objects, such as glass or plastic, leveraging the material identification capability of mmWave radar. This identification result facilitates the robotic arm in modulating its grip force appropriately. Extensive testing reveals that FuseGrasp significantly improves the accuracy of depth reconstruction and material identification for transparent objects. Moreover, real-world robotic trials have confirmed that FuseGrasp markedly enhances the handling of transparent items. A video demonstration of FuseGrasp is available at https://youtu.be/MWDqv0sRSok.

Problem

• Enhance robotic grasping of transparent objects
• Fuse radar and camera for depth completion
• Identify material composition using mmWave radar

Innovation

• Radar-camera fusion enhances transparent object grasping
• Deep neural network fuses radar and camera imagery
• Two-stage training improves depth reconstruction accuracy