Token Merging via Spatiotemporal Information Mining for Surgical Video Understanding

2025-09-28

AI Summary

Vision Transformers (ViTs) incur excessive computational overhead in surgical video understanding due to spatiotemporal token redundancy. Method: This paper proposes a training-free, decoupled token merging approach specifically designed for surgical videos. It introduces a spatiotemporal information-aware framework: temporal merging guided by saliency-weighted aggregation and spatial merging based on temporal stability analysis to dynamically preserve critical regions—thereby jointly preserving temporal continuity and spatial dynamics. The method requires no fine-tuning and is plug-and-play compatible with existing ViT architectures. Contribution/Results: Evaluated across multiple surgical video tasks, the approach achieves over 65% average reduction in GFLOPs while maintaining competitive accuracy. It significantly enhances modeling efficiency for long-sequence surgical videos without compromising performance.

Technology Category

• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Vision Transformer models have shown impressive effectiveness in the surgical video understanding tasks through long-range dependency modeling. However, current methods suffer from prohibitive computational costs due to processing massive spatiotemporal tokens across video frames. While prior work on token merging has advanced model efficiency, they fail to adequately consider the inherent spatiotemporal structure of video data and overlook the heterogeneous nature of information distribution, leading to suboptimal performance. In this paper, we propose a spatiotemporal information mining token merging (STIM-TM) method, representing the first dedicated approach for surgical video understanding. STIM-TM introduces a decoupled strategy that reduces token redundancy along temporal and spatial dimensions independently. Specifically, the temporal component merges spatially corresponding tokens from consecutive frames using saliency weighting, preserving critical sequential information and maintaining continuity. Meanwhile, the spatial component prioritizes merging static tokens through temporal stability analysis, protecting dynamic regions containing essential surgical information. Operating in a training-free manner, STIM-TM achieves significant efficiency gains with over $65%$ GFLOPs reduction while preserving competitive accuracy across comprehensive surgical video tasks. Our method also supports efficient training of long-sequence surgical videos, addressing computational bottlenecks in surgical applications.

Problem

• Reducing computational costs in surgical video understanding
• Merging redundant tokens along temporal and spatial dimensions
• Preserving critical surgical information while improving efficiency

Innovation

• Decoupled token merging along temporal and spatial dimensions
• Saliency weighting merges spatially corresponding tokens temporally
• Temporal stability analysis prioritizes merging static tokens spatially