DIVE: Dynamic Iterative Visual Evidence Construction for Efficient Vision-Language Models

2026-08-05

AI Summary

This work addresses the inefficiency of vision-language models caused by excessively long visual token sequences, a challenge exacerbated by existing single-pass scoring pruning methods that struggle to accurately assess token relevance within contextual interactions. To overcome this limitation, the authors propose a training-free, dynamic iterative pruning framework that reframes token pruning as a process of constructing complementary visual evidence. The approach leverages residual conditional scoring, iterative token selection, and joint updates of visual and prompt residuals to transcend the constraints of one-shot evaluation. Evaluated across eight image understanding benchmarks, the method retains only 11.1% of visual tokens while preserving 98.2% of the original model’s average performance, substantially improving inference efficiency.

Abstract

Visual inputs in vision-language models (VLMs) are often encoded into substantially longer token sequences than text, making visual tokens a major bottleneck for efficient inference. Abundant recent methods address this bottleneck by scoring token importance and pruning low-scoring tokens in a single pass. However, one-shot scoring is insufficient because a token's prompt-relevant usefulness depends on the evidence already retained. Motivated by this insight, we introduce DIVE (Dynamic Iterative Visual Evidence Construction), a training-free framework that recasts visual-token pruning as dynamic evidence construction. DIVE repeatedly selects the remaining token with the highest residual-conditioned score, updates the visual and prompt residuals to discount the evidence already explained, and re-evaluates the remaining tokens. This select-update-re-evaluate process builds a retained set of complementary, prompt-relevant evidence. Experiments across eight image-understanding benchmarks show that DIVE consistently preserves performance across token budgets. With an 88.9% reduction in visual tokens, DIVE retains 98.2% of the uncompressed model's average performance. Code is available at https://github.com/Zhong-Chenchen/DIVE.git.

Problem

• vision-language models
• visual token pruning
• efficient inference
• token importance
• evidence construction

Innovation

• dynamic iterative pruning
• visual evidence construction
• vision-language models
• token reduction
• training-free framework