Streaming 4D Visual Geometry Transformer

2025-07-15

AI Summary

This work addresses the fundamental challenge of real-time, video-driven 4D spatiotemporal geometric reconstruction. We propose the Flow-based 4D Vision Geometry Transformer (F4D-ViT), a novel architecture featuring time-causal attention and a historical key-value caching strategy to enable autoregressive, online streaming inference with implicit memory. Drawing inspiration from efficient attention mechanisms in large language models—such as FlashAttention—and incorporating knowledge distillation from a bidirectional VGGT teacher model, F4D-ViT effectively transfers geometric priors while ensuring computational efficiency. To our knowledge, this is the first successful application of causal Transformers to 4D geometric reconstruction. Evaluated on multiple 4D benchmarks, F4D-ViT achieves real-time online inference speed while maintaining state-of-the-art or competitive reconstruction accuracy. Our approach establishes a new paradigm for scalable, interactive 4D vision systems.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Time-Series/Data Streams

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Perceiving and reconstructing 4D spatial-temporal geometry from videos is a fundamental yet challenging computer vision task. To facilitate interactive and real-time applications, we propose a streaming 4D visual geometry transformer that shares a similar philosophy with autoregressive large language models. We explore a simple and efficient design and employ a causal transformer architecture to process the input sequence in an online manner. We use temporal causal attention and cache the historical keys and values as implicit memory to enable efficient streaming long-term 4D reconstruction. This design can handle real-time 4D reconstruction by incrementally integrating historical information while maintaining high-quality spatial consistency. For efficient training, we propose to distill knowledge from the dense bidirectional visual geometry grounded transformer (VGGT) to our causal model. For inference, our model supports the migration of optimized efficient attention operator (e.g., FlashAttention) from the field of large language models. Extensive experiments on various 4D geometry perception benchmarks demonstrate that our model increases the inference speed in online scenarios while maintaining competitive performance, paving the way for scalable and interactive 4D vision systems. Code is available at: https://github.com/wzzheng/StreamVGGT.

Problem

• Real-time 4D geometry reconstruction from videos
• Efficient streaming long-term 4D perception
• Interactive 4D vision systems with spatial consistency

Innovation

• Causal transformer for online 4D reconstruction
• Temporal attention with cached memory
• Knowledge distillation from dense VGGT