GeoSAM-3D: Geodesic Prompt Propagation for Open-Vocabulary 3D Scene Segmentation from Monocular Video

2026-05-29

AI Summary

This work addresses open-vocabulary 3D scene segmentation under a lightweight setting that relies solely on monocular video, without requiring RGB-D data, multi-view calibration, or explicit mesh reconstruction. The proposed method integrates frozen image and video foundation models with monocular 3D Gaussian splatting reconstruction and introduces a differentiable prompt propagation mechanism based on heat kernel geodesic distance to enable cross-frame 3D mask propagation over the reconstructed scene graph. By innovatively replacing Euclidean nearest neighbors with heat kernel geodesic distance, the approach effectively preserves surface continuity and suppresses inter-object semantic leakage. Experiments demonstrate superior performance across four key aspects: segmentation accuracy, graph-based propagation quality, leakage control, and interactive latency, enabling users to rapidly obtain high-quality 3D segmentations via single-frame clicks or textual queries.

Abstract

Open-vocabulary 3D scene segmentation usually assumes RGB-D video, calibrated multi-view imagery, or a reconstructed mesh. GeoSAM-3D studies a lighter setting: a user uploads a short monocular video, clicks or names an object in one frame, and receives a propagated 3D mask over a Gaussian scene. The implementation combines frozen image and video foundation models with a monocular 3D Gaussian Splatting reconstruction and a differentiable graph-geodesic propagation kernel over Gaussian centroids. The central design choice is to propagate prompts by heat-kernel distance on the reconstructed scene graph, rather than by Euclidean nearest neighbors in 3D. This preserves continuity around curved surfaces and reduces leakage across nearby but disconnected objects. This paper describes the repository state, the mathematical kernel implemented in geosam3d.propagate, the feature head trained from Segment Anything masks, and the validation already present in the codebase. The evaluation protocol separates implementation validation, graph propagation quality, leakage control, and interactive latency.

Problem

• open-vocabulary 3D segmentation
• monocular video
• 3D scene understanding
• prompt propagation
• Gaussian Splatting

Innovation

• geodesic propagation
• 3D Gaussian Splatting
• open-vocabulary segmentation
• monocular video
• heat-kernel distance