ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting

2026-07-21

AI Summary

This work addresses the limitation of existing methods that support only single-target referring expression segmentation and struggle with ambiguous instructions involving an uncertain number of objects—zero, one, or multiple—in real-world scenarios. We introduce, for the first time, the generalized referring 3D Gaussian Splatting segmentation task and establish the GR-LERF and GR-ScanNet benchmarks. The proposed framework requires neither training nor additional feature storage; instead, it leverages multi-view geometric constraints to lift 2D vision-language model priors into 3D space, enabling dynamic, point-level semantic segmentation of arbitrary numbers of targets. Our approach significantly outperforms current state-of-the-art methods in both generalized and single-target settings while maintaining high efficiency, thereby overcoming the conventional reliance on 2D rendering and scene-level optimization in point cloud semantic understanding.

Abstract

Recent advancements in 3D Gaussian Splatting (3DGS) have enabled language-guided scene understanding. However, existing Referring 3D Gaussian Splatting (R3DGS) methods are fundamentally restricted to single-target queries. To reflect the ambiguity of real-world instructions, we introduce the Generalized Referring 3D Gaussian Splatting Segmentation (GR3DGS) task, which requires dynamically segmenting an arbitrary number of targets (0, 1, or $N$). To facilitate comprehensive evaluation of this new task, we construct two new benchmarks: GR-LERF and GR-ScanNet. Crucially, existing R3DGS paradigms exhibit fundamental technical bottlenecks that severely limit their performance on the GR3DGS task: they lack intrinsic 3D point-level understanding by operating merely on 2D rendered pixels, and they incur prohibitive computational overhead by requiring per-scene optimization to embed heavy semantic features. To dismantle these bottlenecks, we propose ZeroSplat, a novel training-free and zero-feature framework. ZeroSplat lifts 2D Vision-Language Model (VLM) priors into 3D space through robust multi-view geometric constraints. This strategy enables intrinsic point-level understanding without incurring any additional feature storage. Extensive experiments demonstrate that ZeroSplat significantly outperforms state-of-the-art methods across generalized and single-target scenarios while maintaining exceptional efficiency. Project Page: https://inkmind-ai.github.io/ZeroSplat

Problem

• Generalized Referring Segmentation
• 3D Gaussian Splatting
• Zero-shot
• Multi-target Segmentation
• 3D Scene Understanding

Innovation

• Generalized Referring Segmentation
• 3D Gaussian Splatting
• Zero-shot Learning
• Vision-Language Model
• Multi-view Geometry