InstanceSplat: Instance-Aware Feed-Forward 3D Gaussian Splatting for Scene Understanding

2026-08-07

AI Summary

This work addresses the limitations of existing feed-forward 3D Gaussian splatting methods, which lack instance awareness, and current instance-aware approaches that rely on per-scene optimization and decouple reconstruction from semantics, hindering joint learning. To overcome these issues, we propose InstanceSplat, the first end-to-end, unified feed-forward 3D Gaussian splatting framework that jointly achieves 3D reconstruction, instance segmentation, and open-vocabulary semantic understanding from pose-free multi-view images in a single forward pass. The key innovation lies in leveraging shared 3D Gaussians to establish cross-view consistent instance features and introducing an instance-guided mechanism for reconstruction and semantic aggregation. InstanceSplat achieves state-of-the-art performance across novel view synthesis, instance segmentation, and semantic understanding, while offering efficient inference, strong generalization, and robustness to unseen scenes and input configurations.

Abstract

Feed-forward 3D Gaussian Splatting (3DGS) enables efficient and generalizable 3D reconstruction, but current feed-forward 3DGS methods for scene understanding remain largely category-oriented. In contrast, instance-aware 3DGS methods typically rely on per-scene optimization and often decouple reconstruction from instance and semantic learning, limiting reciprocal interactions among them. We present InstanceSplat, a unified feed-forward 3DGS framework for generalizable 3D reconstruction and instance-aware scene understanding from pose-free multi-view images. In a single forward pass, InstanceSplat constructs an instance-aware Gaussian representation that jointly encodes appearance, geometry, instance identity, and language-aligned semantics. Shared 3D Gaussians ground instance identities across views, producing renderable and cross-view-consistent instance features. To allow reconstruction and scene understanding to benefit from each other, we further design an instance-centric learning strategy that connects reconstruction, instance learning, and semantic learning through shared instance structure. Specifically, instance cues guide reconstruction, language-aligned semantics strengthen the discrimination of confusing same-category instances, and instance regions aggregate semantic evidence into coherent object-level predictions. Experiments on novel-view synthesis, instance segmentation, and open-vocabulary semantic understanding under varying input-view settings and on an unseen dataset demonstrate state-of-the-art performance, practical efficiency, and strong generalization.

Problem

• 3D Gaussian Splatting
• instance-aware
• scene understanding
• feed-forward
• reconstruction

Innovation

• instance-aware 3D reconstruction
• feed-forward 3D Gaussian Splatting
• cross-view consistency
• language-aligned semantics
• unified scene understanding