Not All Redundant Tokens Are Alike: Analyzing Visual Token Pruning through Token Roles

2026-08-05

AI Summary

This work addresses the computational bottleneck in vision-language models during inference caused by processing a large number of visual tokens, noting that existing pruning methods fail to account for the varying functional roles of redundant tokens. Building on EmbedLens-based token role identification, the study reveals that prevailing pruning strategies implicitly favor certain token roles, yet this preference shows no direct correlation with downstream performance. The authors propose a role-aware pruning strategy that deliberately preserves specific non-“alive” tokens—such as those with weaker semantic alignment—and demonstrate that doing so can maintain or even enhance model performance. These findings underscore the critical influence of functional token roles on importance assessment and offer a novel perspective for designing efficient vision-language models.

Abstract

Vision-language models (VLMs) process an image as a sequence of visual tokens, which creates a substantial computational bottleneck during inference. Recent visual token pruning methods address this issue by removing seemingly redundant tokens, yet it remains unclear how these pruning decisions relate to the functional roles of visual tokens. In this work, we analyze visual token pruning through the lens of token roles identified by EmbedLens. We first show that representative pruning methods exhibit distinct token-role biases, but these biases do not directly correlate with downstream performance. To better understand this behavior, we refine the token-role assignment procedure and evaluate role-protected pruning variants. Our results show that preserving non-alive tokens can sometimes maintain or improve performance, suggesting that tokens with weak direct semantic alignment may still affect model behavior under pruning. Our code is publicly available at https://github.com/jaykim9870/Not_All_Redundant_Tokens_Are_Alike.

Problem

• visual token pruning
• vision-language models
• token roles
• redundant tokens
• computational bottleneck

Innovation

• visual token pruning
• token roles
• vision-language models
• redundancy analysis
• role-protected pruning