CDSeg: A Renderable Gaussian Carrier for Image-to-3D Label Transfer

2026-08-05

AI Summary

This work addresses the challenge of efficiently and accurately transferring 2D image segmentation masks to 3D scenes—such as point clouds or Gaussian-based representations—without requiring task-specific 3D segmentation models. To this end, the authors propose CDSeg, a method that leverages differentiable rendering of 3D Gaussian primitives as semantic carriers. By exploiting pixel-primitive visibility correspondences across multiple views, 2D masks are mapped into 3D space, followed by voting-based fusion and local filtering to achieve robust cross-domain label propagation. The approach unifies support for diverse segmentation paradigms, including semantic, instance, prompt-driven, and LiDAR-based segmentation. Evaluated on DesktopObjects-360, NeRDS-360, and ScanNet-v2, CDSeg achieves mIoU scores of 92.35%, 95.89%, and 65.77%, respectively, while processing million-primitive scenes in under a second.

Abstract

Modern image models provide strong cues about \emph{what} should be segmented in each view, but their masks do not by themselves determine \emph{where} those labels should persist in 3D. We present Cross-Domain Segmentation via Gaussian Splatting (CDSeg), a label-transfer interface that requires no task-specific 3D segmentation training and uses Gaussian primitives as a renderable label carrier. An external mask source supplies the labels, while renderer-derived visibility determines which 3D primitives receive them. The carrier is instantiated either by completing each input point into one Gaussian, preserving its index, or by reusing the native primitives of an optimized Gaussian scene. CDSeg records pixel--primitive associations during rendering and fuses multi-view masks through voting and a local filter. The resulting labels can be returned to the original points, retained on the native Gaussian scene, or rendered into other views. CDSeg covers promptable, automatic instance, semantic, and LiDAR settings and processes scenes with millions of primitives in seconds. It obtains 92.35\% mIoU on DesktopObjects-360, 95.89\% on NeRDS-360, and 65.77\% on the full ScanNet-v2 validation split using the provided 2D semantic annotations. CDSeg thereby provides one interface for reusing 2D masks across point clouds, Gaussian scenes, and image views without a task-specific 3D segmentation network.

Problem

• 3D label transfer
• image-to-3D segmentation
• Gaussian splatting
• cross-domain segmentation
• renderable label carrier

Innovation

• Gaussian Splatting
• Label Transfer
• 3D Segmentation
• Cross-Domain
• Renderable Primitives