Sketchy Bounding-box Supervision for 3D Instance Segmentation

2025-05-22

AI Summary

To address the challenge of acquiring precise 3D bounding box annotations for weakly supervised 3D instance segmentation, this paper introduces a novel weakly supervised paradigm leveraging inaccurate “sketch bounding boxes”—ground-truth boxes perturbed by scaling, translation, and rotation. Methodologically, we propose the first formal modeling of sketch bounding boxes; design an adaptive box-to-point pseudo-labeler that accurately assigns overlapping point cloud regions; and establish a coarse-to-fine joint optimization framework enabling iterative co-training of the pseudo-label generator and the segmenter. Evaluated on ScanNetV2 and S3DIS, our method achieves state-of-the-art performance using only sketch boxes—surpassing multiple fully supervised approaches. This demonstrates an exceptional trade-off between annotation efficiency and segmentation accuracy, significantly reducing labeling cost while maintaining high fidelity.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Semi-Supervised Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Bounding box supervision has gained considerable attention in weakly supervised 3D instance segmentation. While this approach alleviates the need for extensive point-level annotations, obtaining accurate bounding boxes in practical applications remains challenging. To this end, we explore the inaccurate bounding box, named sketchy bounding box, which is imitated through perturbing ground truth bounding box by adding scaling, translation, and rotation. In this paper, we propose Sketchy-3DIS, a novel weakly 3D instance segmentation framework, which jointly learns pseudo labeler and segmentator to improve the performance under the sketchy bounding-box supervisions. Specifically, we first propose an adaptive box-to-point pseudo labeler that adaptively learns to assign points located in the overlapped parts between two sketchy bounding boxes to the correct instance, resulting in compact and pure pseudo instance labels. Then, we present a coarse-to-fine instance segmentator that first predicts coarse instances from the entire point cloud and then learns fine instances based on the region of coarse instances. Finally, by using the pseudo instance labels to supervise the instance segmentator, we can gradually generate high-quality instances through joint training. Extensive experiments show that our method achieves state-of-the-art performance on both the ScanNetV2 and S3DIS benchmarks, and even outperforms several fully supervised methods using sketchy bounding boxes. Code is available at https://github.com/dengq7/Sketchy-3DIS.

Problem

• Addresses inaccurate bounding boxes in 3D instance segmentation
• Proposes adaptive pseudo labeler for overlapping sketchy boxes
• Develops coarse-to-fine segmentator for high-quality instance prediction

Innovation

• Uses sketchy bounding boxes with perturbations
• Jointly trains pseudo labeler and segmentator
• Adaptive box-to-point pseudo labeler