HTTM: Head-wise Temporal Token Merging for Faster VGGT

2025-11-26

AI Summary

VGGT suffers from severe inference latency in large-scale 3D scene reconstruction due to global attention requiring all-pairwise computations over view tokens, especially for long view sequences. Existing token merging methods uniformly compress tokens across attention heads, leading to feature homogenization and degraded representational capacity. To address this, we propose a training-free, head-level temporal token merging method grounded in the intrinsic architecture of multi-head attention. Leveraging the spatial locality and temporal correspondence inherent to individual attention heads, our approach performs heterogeneous token merging at the head level—preserving feature uniqueness without model fine-tuning. This significantly shortens attention sequence lengths while maintaining architectural compatibility. Experiments demonstrate up to 7× GPU inference speedup with negligible degradation in reconstruction accuracy, substantially enhancing efficiency for large-scene, long-sequence 3D reconstruction.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

The Visual Geometry Grounded Transformer (VGGT) marks a significant leap forward in 3D scene reconstruction, as it is the first model that directly infers all key 3D attributes (camera poses, depths, and dense geometry) jointly in one pass. However, this joint inference mechanism requires global attention layers that perform all-to-all attention computation on tokens from all views. For reconstruction of large scenes with long-sequence inputs, this causes a significant latency bottleneck. In this paper, we propose head-wise temporal merging (HTTM), a training-free 3D token merging method for accelerating VGGT. Existing merging techniques merge tokens uniformly across different attention heads, resulting in identical tokens in the layers' output, which hinders the model's representational ability. HTTM tackles this problem by merging tokens in multi-head granularity, which preserves the uniqueness of feature tokens after head concatenation. Additionally, this enables HTTM to leverage the spatial locality and temporal correspondence observed at the head level to achieve higher merging ratios with lower merging costs compared to existing methods. Thus, HTTM achieves up to 7x acceleration with negligible performance drops in a GPU-based inference.

Problem

• Accelerates VGGT by merging tokens per attention head
• Preserves token uniqueness after head concatenation in transformers
• Enables higher merging ratios with lower computational costs

Innovation

• Head-wise temporal merging accelerates VGGT
• Merges tokens at multi-head granularity
• Leverages spatial locality for higher merging ratios