VETO: Video Efficient Token Optimization for Vision Language Models

2026-10-01

AI Summary

This study addresses the prohibitive inference costs in long video processing caused by the quadratic growth of visual tokens, as well as the efficiency bottlenecks inherent in existing single-axis compression methods. To overcome these limitations, this work proposes VETO, a plug-and-play module that introduces a novel "spatial-before-temporal" hierarchical compression mechanism. By leveraging optimal transport matching, VETO achieves dual-axis collaborative compression through intra-frame semantic merging and inter-frame redundancy elimination, making it fully compatible with full-attention architectures. The proposed method surpasses the performance ceiling of single-axis approaches, accelerating inference by 45%. Under extreme token budgets, it attains an accuracy of 55.7%, outperforming mainstream baselines. Furthermore, VETO consistently preserves or enhances zero-shot performance across multiple foundation models, demonstrating its broad applicability and effectiveness for efficient long video understanding.

Abstract

Processing long videos with Vision-Language Models (VLMs) is bottlenecked by the quadratic cost of visual tokens, making long-form inference prohibitively expensive. While single-axis compression methods mitigate this, they hit a hard efficiency floor because they treat spatial and temporal redundancy independently. We present VETO (Video Efficient Token Optimization for Vision-Language Models), a training-optional plug-in that eliminates this bottleneck through dual-axis compression: (i) an intra-frame compressor that merges semantically similar tokens within each frame via optimal-transport inspired matching, and (ii) an inter-frame compressor that identifies and merges temporally redundant frames. The key design insight is hierarchical ordering: by first compressing spatial dimensions, VETO drastically reduces the cost of subsequent global temporal matching, bypassing the efficiency wall of single-axis approaches, with an advantage that grows with modern fully-fused attention infrastructure. Empirically, VETO achieves up to 45% faster inference (e.g., on LLaVA-OneVision-7B) while preserving or improving accuracy. Under extreme token starvation (10% budget), VETO outperforms VFlowOpt (54.9%), VisionZip (52.6%), and FastV (47.9%) with 55.7% accuracy. We demonstrate universal applicability across LLaVA-OneVision, InternVL-2.5, and LongVA, with zero-shot accuracy preserved or improved in all cases.

Problem

• Vision-Language Models
• long video processing
• visual token redundancy
• computational efficiency
• spatiotemporal compression

Innovation

• Dual-axis Compression
• Optimal Transport
• Training-optional Plug-in
• Hierarchical Ordering
• Vision Language Models