From Token Importance to Conditional Removability: Rethinking Visual Token Pruning in Multimodal Large Language Models

2026-09-22

AI Summary

本文针对视觉令牌修剪问题，提出CoRePrune框架，通过两阶段方法考虑删除上下文和表示深度对令牌可移除性的影响，有效保持模型性能。

Abstract

Training-free visual-token pruning often uses token importance, redundancy, or related selection criteria as proxies for safe removal. We show that these signals alone do not fully characterize removability, which is conditioned on both representation depth and the surrounding deletion set. Controlled interventions demonstrate that removing the same tokens at different depths produces substantially different downstream perturbations, while changing only the deletion context at a fixed depth alters candidate marginals and pruning-boundary decisions. These findings show that token importance alone cannot determine when a token is safely removable or how its removability changes under joint deletion. Motivated by this perspective, we propose CoRePrune, a training-free two-stage framework. Progressive Perturbation-Aware Visual Pruning refreshes deletion effects as visual representations evolve, while Set-Conditioned Refinement reevaluates candidate rescue benefits under the current deletion set after visual--text interaction. Across five multimodal large language model backbones covering standard images, high-resolution inputs, and video, CoRePrune preserves performance under aggressive token budgets. On Qwen3.5, with a final budget of 128 visual tokens, it retains 90.3% of dense-model performance while reducing aggregate prefill time by 51.0%.

Problem

• visual-token pruning
• representation depth
• deletion context

Innovation

• Conditional Removability
• Progressive Perturbation-Aware Visual Pruning
• Set-Conditioned Refinement