IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer

2026-07-21

AI Summary

This work addresses the challenge of achieving temporally consistent modeling of both geometric structure and object identity in dynamic scenes with existing streaming methods. The authors propose a streaming 4D Instance-Anchored Geometry Transformer that processes video frames online through causal spatiotemporal modeling, incrementally updating a unified representation of camera poses, geometry, and instance identities. Key contributions include the first unified online learning framework for geometry and instance semantics over long-term dynamic sequences, the construction of InsScene4D-147K—a large-scale temporally consistent 4D dataset—and the design of a causal spatiotemporal Transformer, an incremental geometry-instance joint representation, and a geometry-guided automatic annotation pipeline. The method significantly outperforms current streaming approaches in 3D reconstruction, pose estimation, instance tracking, and open-vocabulary segmentation, enabling scalable long-term online inference.

Abstract

Real-world spatial intelligence requires agents to understand scenes from continuous video streams, where objects move, persist, disappear, and reappear over time. While recent spatial foundation models have enabled generalizable feed-forward 3D reconstruction, most streaming methods remain geometry-centric and lack temporally consistent object-level understanding. Meanwhile, existing semantic reconstruction and 3D-aware vision-language methods largely rely on externally extracted 2D semantic cues or loosely coupled geometry inputs, limiting unified geometry-instance learning in long dynamic scenes. In this paper, we propose IGGT4D, a streaming instance-grounded geometry Transformer for online 4D scene understanding. IGGT4D processes video frames sequentially, reuses historical context through causal spatial-temporal modeling, and incrementally updates a unified representation of camera motion, geometry, and object identity. This enables long-sequence feed-forward reconstruction with geometry-instance consistency in dynamic environments. To address the lack of high-quality 4D supervision, we further construct InsScene4D-147K, a large-scale dataset spanning real/synthetic and static/dynamic scenes, with RGB images, depth, poses, and temporally consistent instance masks generated by an automated geometry-guided annotation pipeline. Experiments on 3D reconstruction, pose estimation, instance spatial tracking, and open-vocabulary segmentation demonstrate that IGGT4D outperforms existing streaming baselines while maintaining scalable online inference for long dynamic sequences.

Problem

• 4D scene understanding
• temporal consistency
• instance-level understanding
• geometry-instance learning
• streaming reconstruction

Innovation

• streaming 4D reconstruction
• instance-grounded geometry
• causal spatial-temporal modeling
• unified geometry-instance representation
• online inference