OccGS: Zero-shot 3D Occupancy Reconstruction with Semantic and Geometric-Aware Gaussian Splatting

2025-02-07

AI Summary

Zero-shot semantic 3D occupancy reconstruction from unlabeled multi-sensor data remains challenging due to the absence of semantic supervision and effective joint geometric-semantic modeling. Method: This paper proposes a scene-aware geometry-semantic joint reconstruction framework. Its core components include: (1) a novel Gaussian lattice modeling paradigm that jointly perceives semantics and geometry; (2) a cumulative Gaussian voxel splatting algorithm enabling efficient, differentiable voxelization; and (3) integration of vision-language model priors, LiDAR-guided geometric constraints, and jointly parameterized Gaussian representations—entirely without semantic annotations. Results: Under zero-shot settings, our method achieves state-of-the-art performance, substantially outperforming existing self-supervised approaches while attaining semantic occupancy accuracy close to fully supervised baselines. It establishes a new paradigm for open-world 3D understanding.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Obtaining semantic 3D occupancy from raw sensor data without manual annotations remains an essential yet challenging task. While prior works have approached this as a perception prediction problem, we formulate it as scene-aware 3D occupancy reconstruction with geometry and semantics. In this work, we propose OccGS, a novel 3D Occupancy reconstruction framework utilizing Semantic and Geometric-Aware Gaussian Splatting in a zero-shot manner. Leveraging semantics extracted from vision-language models and geometry guided by LiDAR points, OccGS constructs Semantic and Geometric-Aware Gaussians from raw multisensor data. We also develop a cumulative Gaussian-to-3D voxel splatting method for reconstructing occupancy from the Gaussians. OccGS performs favorably against self-supervised methods in occupancy prediction, achieving comparable performance to fully supervised approaches and achieving state-of-the-art performance on zero-shot semantic 3D occupancy estimation.

Problem

• Zero-shot 3D occupancy reconstruction
• Semantic and geometric-aware Gaussian splatting
• Raw sensor data without manual annotations

Innovation

• Zero-shot 3D occupancy reconstruction
• Semantic and Geometric-Aware Gaussian Splatting
• Cumulative Gaussian-to-3D voxel splatting