🤖 AI Summary
To address the prohibitive inference cost of large vision-language models (VLMs) under long-context, multi-image inputs, this paper proposes an adaptive visual token pruning method. Unlike prior approaches, it uniquely decouples redundancy modeling into two orthogonal dimensions: intra-image diversity and inter-image dissimilarity, and introduces a Pareto-optimal selection mechanism to jointly optimize both against text alignment. The proposed two-stage framework—comprising global candidate pool construction, diversity quantification, and greedy subset selection—dynamically allocates token budgets and identifies the most representative visual tokens without fine-tuning or supervision. Evaluated on multi-image long-context benchmarks, our method reduces visual tokens by up to 67% while maintaining or even improving performance on VQA and image captioning tasks. This achieves content-aware, cross-image cooperative inference with significant efficiency gains.
📝 Abstract
Large Multimodal Models (LMMs) have proven effective on various tasks. They typically encode visual inputs into Original Model sequences of tokens, which are then concatenated with textual tokens and jointly processed by the language model. However, the growing number of visual tokens greatly increases inference cost. Visual token pruning has emerged as a promising solution. However, existing methods often overlook scenarios involving long context inputs with multiple images. In this paper, we analyze the challenges of visual token pruning in long context, multi-image settings and introduce an adaptive pruning method tailored for such scenarios. We decompose redundancy into intra-image and inter-image components and quantify them through intra-image diversity and inter-image variation, which jointly guide dynamic budget allocation. Our approach consists of two stages. The intra-image stage allocates each image a content-aware token budget and greedily selects its most representative tokens. The inter-image stage performs global diversity filtering to form a candidate pool and then applies a Pareto selection procedure that balances diversity with text alignment. Extensive experiments show that our approach maintains strong performance in long context settings while significantly cutting down the number of visual tokens.