Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors

2026-06-05

AI Summary

This work addresses the limitation of existing 3D multimodal large models, which rely on offline, complete scene inputs and thus struggle to support real-time 3D spatial understanding in streaming video. The authors propose the first online 3D vision-language model capable of processing streaming inputs, leveraging autoregressive streaming control modeling, a Visual-Spatial Feature Integration (VSFI) module, and a Geometry-Adaptive Voxel Compression (GAVC) mechanism to enable efficient real-time 3D understanding and long-context modeling. The study also introduces the first large-scale online 3D spatiotemporal question-answering dataset and a comprehensive benchmark covering 29 tasks. Experiments demonstrate that the proposed method significantly outperforms both open-source and closed-source models in online and offline 3D spatial reasoning and localization tasks, validating its effectiveness and generalization capability.

Abstract

Despite advances in 3D scene understanding, existing 3D Large Multimodal Models operate in offline settings, requiring complete scene observations or predefined video clips. In this paper, we present an online 3D vision-language model that enables real-time spatial understanding from streaming video. Our approach adopts an autoregressive streaming control modeling based on the LLM's next-token prediction objective to learn when to respond, and employs a lightweight Visual-Spatial Feature Integration (VSFI) module to incrementally inject temporally aligned geometry priors into the visual stream. To alleviate long-context decoding overhead, we propose a plug-and-play Geometry-Adaptive Voxel Compression (GAVC) module for efficient visual token compression. To address the scarcity of streaming 3D-language data, we further develop a scalable data generation pipeline that curates over 1M online spatio-temporal 3D QA pairs and establishes a comprehensive benchmark spanning 29 tasks. Extensive experiments show that our approach significantly outperforms both proprietary and open-source models across online and offline 3D spatial understanding, reasoning, and grounding tasks. The project page is available at https://stream3d-vlm.github.io/

Problem

• online 3D understanding
• streaming video
• 3D vision-language model
• real-time spatial reasoning

Innovation

• online 3D vision-language model
• incremental geometry priors
• streaming control modeling
• Visual-Spatial Feature Integration (VSFI)
• Geometry-Adaptive Voxel Compression (GAVC)