InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models

2026-06-01

AI Summary

This work addresses the high computational cost of video large language models caused by excessive visual tokens, a challenge exacerbated by the sensitivity of existing training-free compression methods to frame-level noise and their inability to handle uneven information distribution across videos. To overcome these limitations, the authors propose InfoMerge, a training-free visual token compression approach that models spatiotemporal similarity through segment-level second-order temporal redundancy estimation—termed Temporal Fingerprint Difference—and dynamically optimizes token allocation via a content-aware budget assignment (CABA) mechanism based on clip distinctiveness and spectral entropy. Evaluated on LLaVA-OneVision-7B, InfoMerge retains 98.8% of the original performance using only 15% of visual tokens and achieves a 4.24× speedup during the prefill phase, demonstrating superior efficiency–accuracy trade-offs across multiple benchmarks and backbone architectures.

Abstract

Video Large Language Models (Video-LLMs) achieve strong performance in video understanding, but their excessive visual tokens bring substantial computational overhead. Existing training-free compression methods improve inference efficiency by reducing visual tokens, yet they often rely on local adjacent-frame similarity for temporal redundancy estimation or allocate token budgets mainly according to segment length. Such designs are sensitive to frame-level noise and fail to capture the non-uniform information distribution of real-world videos. To address these challenges, we propose InfoMerge, a training-free visual token compression method that improves token utilization through robust redundancy estimation and content-aware budget allocation. Specifically, we propose the Temporal Fingerprint Difference: a segment-level second-order temporal redundancy estimation strategy, which models the temporal similarity structure of tokens at the same spatial positions within each segment. We further introduce Content-Aware Budget Allocation (CABA), which dynamically allocates segment-level token budgets based on segment uniqueness and spectral-entropy-based representational richness. By reducing repeated preservation of redundant static regions and allocating more tokens to informative segments, InfoMerge makes better use of the limited token budget while maintaining strong performance. Extensive experiments show that InfoMerge achieves strong efficiency--accuracy trade-offs across multiple benchmarks and backbones, with more pronounced advantages under aggressive compression. On LLaVA-OneVision-7B, InfoMerge retains 98.8\% of the original average performance while reducing 85\% of visual tokens and achieving a 4.24-fold speedup in the prefill stage.

Problem

• Video Large Language Models
• token compression
• temporal redundancy
• information distribution
• computational overhead

Innovation

• Temporal Fingerprint Difference
• Content-Aware Budget Allocation
• Token Compression
• Video Large Language Models
• Training-Free Compression