IAM: Enhancing RGB-D Instance Segmentation with New Benchmarks

2025-01-03

AI Summary

RGB-D instance segmentation suffers from weak fine-grained object discrimination and a severe scarcity of high-quality, instance-level annotations. Method: We introduce three high-precision, instance-annotated indoor RGB-D benchmark datasets—the first dedicated benchmark suite for RGB-D instance segmentation—and propose a lightweight, robust cross-modal feature fusion framework integrating multimodal alignment, dual-stream encoder collaboration, and instance-aware mask decoding. Contribution/Results: Our method achieves an 8.2% mAP improvement on the new benchmarks, significantly enhancing fine-grained segmentation accuracy—particularly for small objects—and improving model generalization. The proposed datasets and method jointly advance reliable scene understanding and manipulation capabilities for service robotics applications.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Segmentation
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health

Abstract

Image segmentation is a vital task for providing human assistance and enhancing autonomy in our daily lives. In particular, RGB-D segmentation-leveraging both visual and depth cues-has attracted increasing attention as it promises richer scene understanding than RGB-only methods. However, most existing efforts have primarily focused on semantic segmentation and thus leave a critical gap. There is a relative scarcity of instance-level RGB-D segmentation datasets, which restricts current methods to broad category distinctions rather than fully capturing the fine-grained details required for recognizing individual objects. To bridge this gap, we introduce three RGB-D instance segmentation benchmarks, distinguished at the instance level. These datasets are versatile, supporting a wide range of applications from indoor navigation to robotic manipulation. In addition, we present an extensive evaluation of various baseline models on these benchmarks. This comprehensive analysis identifies both their strengths and shortcomings, guiding future work toward more robust, generalizable solutions. Finally, we propose a simple yet effective method for RGB-D data integration. Extensive evaluations affirm the effectiveness of our approach, offering a robust framework for advancing toward more nuanced scene understanding.

Problem

• RGB-D Object Segmentation
• Fine-grained Detail Discrimination
• Robot Navigation and Grasping

Innovation

• RGB-D Object Segmentation
• Color-Depth Integration
• Improved Scene Understanding