RegVGGT: Sustainable Visual Geometry Grounding for Streaming via Regulated Memory

2026-09-19

AI Summary

针对长时间视频流3D重建中GPU内存限制问题，提出RegVGGT方法通过控制每帧最多1%的token更新上下文内存，有效抑制内存膨胀并保持高质量重建。

Abstract

3D reconstruction from a lengthy video stream input poses a dilemma for feed-forward reconstruction models (FFRMs), that a whole-stream inference context cannot be retained under limited GPU memory.Recent studies seek to resolve this problem via a trade-off between the integrity of inference context and GPU memory usage, which either suffer from a rapid memory inflation or degraded context integrity due to artificially capping memory usage.Driven by our key observation that the initial saliency of a token reliably dictates its long-term importance across the stream, we propose RegVGGT, a training-free token regulation method which aggressively regulates the tokens of incoming frames.By admitting at most 1% of tokens per frame to update the context memory, our method dramatically suppresses memory inflation as the stream progresses.Equipped with a FlashAttention-compatible token saliency estimation scheme, RegVGGT is capable of processing thousands of frames on a consumer-grade GPU with negligible compromise to reconstruction quality.Extensive experiments demonstrate that RegVGGT achieves state-of-the-art performance on long-horizon benchmarks across diverse FFRM prediction tasks, surpassing prior FFRM-based stream reconstruction baselines by a large margin.

Problem

• 3D reconstruction
• video stream
• GPU memory
• inference context

Innovation

• token regulation
• memory inflation control
• FlashAttention-compatible
• saliency estimation