ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understanding

2025-12-03

AI Summary

Existing Gaussian methods for 3D scene understanding suffer from two key limitations: closed-set semantic modeling neglects rendering fidelity, while purely 2D self-supervised approaches compromise geometric quality and generalization. This paper introduces ShelfGaussian—the first Gaussian framework supporting open-vocabulary semantics and multimodal joint optimization. Its core contributions are threefold: (1) a Multimodal Gaussian Transformer that fuses Gaussian radiance fields with vision foundation model features; (2) a “shelf-style” supervision paradigm enabling synergistic optimization of 2D image reconstruction and 3D geometry; and (3) a cross-sensor feature alignment and querying mechanism unifying open-vocabulary semantic modeling with high-fidelity rendering. ShelfGaussian achieves state-of-the-art zero-shot semantic occupancy prediction on Occ3D-nuScenes and demonstrates robust野外 perception and planning capabilities on a real-world unmanned ground vehicle (UGV) platform.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

We introduce ShelfGaussian, an open-vocabulary multi-modal Gaussian-based 3D scene understanding framework supervised by off-the-shelf vision foundation models (VFMs). Gaussian-based methods have demonstrated superior performance and computational efficiency across a wide range of scene understanding tasks. However, existing methods either model objects as closed-set semantic Gaussians supervised by annotated 3D labels, neglecting their rendering ability, or learn open-set Gaussian representations via purely 2D self-supervision, leading to degraded geometry and limited to camera-only settings. To fully exploit the potential of Gaussians, we propose a Multi-Modal Gaussian Transformer that enables Gaussians to query features from diverse sensor modalities, and a Shelf-Supervised Learning Paradigm that efficiently optimizes Gaussians with VFM features jointly at 2D image and 3D scene levels. We evaluate ShelfGaussian on various perception and planning tasks. Experiments on Occ3D-nuScenes demonstrate its state-of-the-art zero-shot semantic occupancy prediction performance. ShelfGaussian is further evaluated on an unmanned ground vehicle (UGV) to assess its in the-wild performance across diverse urban scenarios. Project website: https://lunarlab-gatech.github.io/ShelfGaussian/.

Problem

• Develops open-vocabulary 3D scene understanding using Gaussian representations
• Integrates multi-modal sensor data via a Gaussian Transformer for feature querying
• Optimizes Gaussians with off-the-shelf vision models at 2D and 3D levels

Innovation

• Multi-Modal Gaussian Transformer for cross-sensor feature querying
• Shelf-Supervised Learning with vision foundation models at 2D and 3D levels
• Open-vocabulary 3D scene understanding using Gaussian-based representations