EvoCut: Multi-Layer Evolution-Aware Visual Token Compression for Efficient Large Vision-Language Models

2026-06-01

AI Summary

This work addresses the inefficiency of current vision-language models caused by the excessive number of visual tokens generated by their vision encoders. Conventional token compression methods, which assess token importance based solely on single-layer attention or representations, overlook the dynamic evolution of tokens across multiple layers, leading to significant performance degradation. To overcome this limitation, the paper proposes a training-free, attention-free visual token compression method that introduces, for the first time, multi-layer evolutionary deviation as a measure of token importance. By analyzing the trajectory of each token’s representation across layers and preserving those that deviate from the collective trend—indicating high information content—the method achieves remarkable efficiency gains. On LLaVA-1.5-7B, it retains only 11.1% of visual tokens while preserving 94.4% of average performance, substantially accelerating inference without compromising accuracy.

Abstract

Large vision-language models (LVLMs) achieve strong performance on image and video understanding tasks, but their inference efficiency is constrained by the large number of visual tokens produced by vision encoders. Most existing visual token compression methods estimate token importance from attention scores or representation properties at specific layers, overlooking how visual tokens evolve across the vision encoder. Such layer-specific criteria may provide incomplete importance estimates and limit performance preservation after compression. To address this issue, we analyze layer-wise visual token evolution directions and observe that tokens form multiple group evolution directions across vision-encoder layers. Our analysis further shows that informative tokens tend to exhibit persistent deviations from common group evolution directions. Based on this observation, we propose EvoCut, a training-free and attention-free visual token compression method that estimates token importance from multi-layer evolution deviation. Experimental results show that EvoCut can retain only 11.1\% of the visual tokens on LLaVA-1.5-7B while preserving 94.4\% of the average performance, demonstrating its effectiveness in balancing efficiency and accuracy.

Problem

• visual token compression
• vision-language models
• token importance
• layer-wise evolution
• inference efficiency

Innovation

• visual token compression
• multi-layer evolution
• evolution deviation
• training-free
• large vision-language models