CutS3D: Cutting Semantics in 3D for 2D Unsupervised Instance Segmentation

2024-11-25

arXiv.org

AI Summary

Unsupervised 2D instance segmentation struggles with disentangling overlapping objects, as existing approaches model only semantic information and lack spatial decoupling capability. To address this, we propose the first point-cloud-based 3D semantic cutting paradigm: leveraging scene-level point clouds to construct a geometry-aware 3D spatial representation, projecting 2D semantic masks into 3D space and decoupling them into precise instances. We design a spatial importance function to enhance boundary semantics in 3D and introduce a three-component spatial confidence mechanism to mitigate ambiguity in pseudo-labels. Our method integrates point-cloud geometric modeling, joint training with a class-agnostic detector, and pseudo-label confidence refinement. Evaluated on PASCAL VOC and COCO benchmarks, our approach significantly outperforms state-of-the-art unsupervised instance segmentation and object detection methods.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Semi-Supervised Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Traditionally, algorithms that learn to segment object instances in 2D images have heavily relied on large amounts of human-annotated data. Only recently, novel approaches have emerged tackling this problem in an unsupervised fashion. Generally, these approaches first generate pseudo-masks and then train a class-agnostic detector. While such methods deliver the current state of the art, they often fail to correctly separate instances overlapping in 2D image space since only semantics are considered. To tackle this issue, we instead propose to cut the semantic masks in 3D to obtain the final 2D instances by utilizing a point cloud representation of the scene. Furthermore, we derive a Spatial Importance function, which we use to resharpen the semantics along the 3D borders of instances. Nevertheless, these pseudo-masks are still subject to mask ambiguity. To address this issue, we further propose to augment the training of a class-agnostic detector with three Spatial Confidence components aiming to isolate a clean learning signal. With these contributions, our approach outperforms competing methods across multiple standard benchmarks for unsupervised instance segmentation and object detection.

Problem

• Separate overlapping 2D instances using 3D semantics
• Improve pseudo-masks with Spatial Importance function
• Enhance detector training with Spatial Confidence components

Innovation

• Utilize 3D point cloud for 2D segmentation
• Introduce Spatial Importance for 3D borders
• Augment training with Spatial Confidence components