MonoVoc: Decoupling Geometry and Semantics for Lightweight Monocular Open-Vocabulary 3D Gaussians

2026-07-30

AI Summary

This work addresses the limitations of existing 3D Gaussian-based methods, which rely on multi-view inputs and scene-specific optimization while incurring substantial memory overhead due to dense storage of language features, thereby hindering efficient open-vocabulary 3D semantic understanding. The authors propose a lightweight, training-free pipeline that decouples monocular video input into two stages: geometric reconstruction followed by semantic integration. It first recovers 3D geometry independently and then injects object-level semantics via modular post-processing. This approach achieves the first explicit disentanglement of geometry and semantics within an open-vocabulary 3D Gaussian framework, replacing per-Gaussian language embeddings with object-level representations to drastically reduce memory consumption and enhance scalability. On the Replica dataset, the method maintains high-quality rendering and segmentation accuracy while reducing memory usage by an order of magnitude compared to state-of-the-art approaches, enabling efficient open-vocabulary 3D retrieval and question answering.

Abstract

Open vocabulary 3D scene understanding is essential for next-generation interactive systems, empowering users to intuitively query and navigate reconstructed environments using natural language. However, current 3D Gaussian frameworks are often bottlenecked by restrictive multiview capture requirements, costly scene-specific optimization, and the massive memory overhead of storing dense language features. We present a novel, training-free pipeline that fundamentally reimagines this paradigm by explicitly decoupling 3D geometric reconstruction from semantic integration. Given a standard monocular video sequence as input, our method efficiently outputs a compact, highly interpretable, and fully searchable object-level semantic Gaussian map. Rather than entangling heavy language embeddings within the mapping loop, we extract geometry independently and ground semantics through a lightweight, modular post-processing framework. Extensive evaluations on the Replica dataset demonstrate that this decoupled architecture preserves strong rendering fidelity and competitive segmentation accuracy. Crucially, by replacing dense per-Gaussian storage with modular, object-level semantic embeddings, our approach delivers an order-of-magnitude reduction in memory usage compared to SOTA baselines. This provides a highly efficient, scalable, and practical solution for open-vocabulary 3D retrieval and question answering directly from everyday monocular video.

Problem

• open-vocabulary 3D understanding
• monocular 3D reconstruction
• memory overhead
• semantic-geometric coupling
• language-grounded 3D retrieval

Innovation

• decoupling geometry and semantics
• monocular open-vocabulary 3D
• lightweight semantic Gaussian mapping
• object-level semantic embeddings
• training-free 3D reconstruction