CDIS: Cross-Dimensional Class-Agnostic 3D Instance Segmentation via 2D Mask Tracking and 3D-2D Projection Merging

2026-07-20

AI Summary

This work addresses the challenges of category-agnostic 3D instance segmentation in unknown environments, where frame-by-frame projection often leads to identity fragmentation and spatial discontinuities. To overcome these limitations without requiring any 3D training data, the authors propose a zero-shot method that establishes a cross-dimensional feedback loop between 2D and 3D representations. Specifically, 2D instance masks are tracked across frames and associated with 3D superpoints, thereby integrating temporally stable 2D trajectories with spatially coherent 3D regions. This fusion enables, for the first time, the generation of globally consistent 3D instance labels in a zero-shot setting. The proposed approach significantly outperforms existing zero-shot methods across multiple benchmarks, demonstrating superior performance in terms of accuracy, temporal consistency, and scalability.

Abstract

Class-agnostic 3D instance segmentation is critical for robotic systems operating in unknown environments, enabling perception of previously unseen objects for reliable manipulation and navigation. Existing approaches typically project per-frame 2D instance masks into 3D and merge them, which often breaks object identities across time and yields fragmented 3D instances. We introduce Cross-Dimensional Class-Agnostic 3D Instance Segmentation (CDIS), a zero-shot framework that explicitly tracks 2D instance masks across frames and associates them with 3D superpoints, creating a feedback loop between 2D and 3D. This cross-dimensional reasoning links temporally stable 2D tracks with spatially coherent 3D regions, producing globally consistent 3D instance labels without any 3D-specific training. Experiments on benchmark datasets demonstrate that CDIS achieves higher accuracy and consistency than state-of-the-art zero-shot methods, while remaining efficient and scalable to diverse real-world environments.

Problem

• class-agnostic
• 3D instance segmentation
• 2D mask tracking
• 3D-2D projection
• object identity

Innovation

• class-agnostic
• 3D instance segmentation
• 2D mask tracking
• cross-dimensional reasoning
• zero-shot