Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models

2026-06-10

AI Summary

This work addresses the high computational cost and KV cache overhead in vision-language models caused by the large number of visual tokens, as well as the performance degradation induced by existing removal-based compression methods that irreversibly discard critical information. The authors propose Reroute, a training-free, plug-in mechanism that temporarily retains pruned visual tokens and enables their reactivation in subsequent layers through recoverable routing, without altering scoring rules or computational budgets. By integrating staged attention scoring with dynamic routing scheduling, Reroute is compatible with mainstream pruning strategies such as FastV, PDrop, and Nüwa. Experiments on LLaVA-1.5 and Qwen backbones demonstrate that Reroute significantly enhances referential comprehension under aggressive compression while preserving general visual question answering performance.

Abstract

Vision-language models (VLMs) project images into hundreds to thousands of visual tokens, making decoder inference expensive in both attention computation and KV-cache memory. Existing visual-token reduction methods largely follow a rank-and-remove paradigm: they score visual tokens, keep a compact subset, and permanently discard the rest. We show that this irreversible action is fragile because visual-token importance changes across decoder depth; tokens ranked low at one stage may become relevant in later layers, especially for grounding-sensitive queries. We propose Reroute, a training-free plug-in that replaces removal with recoverable routing. At each routing stage, selected vision tokens pass through decoder blocks, while deferred tokens bypass the stage and re-enter the candidate pool at the next routing decision. Reroute reuses existing attention-score ranking rules and stage-wise schedules, preserving the theoretical TFLOPs and KV-cache budget class of the pruning method it augments. Across FastV, PDrop, and Nüwa variants on LLaVA-1.5 and Qwen backbones, reroute improves grounding under aggressive token reduction while maintaining general VQA performance. These results suggest that VLM token reduction should not be viewed only as irreversible pruning, but also as recoverable routing. The code can be found here: https://github.com/elmma/mllm-reroute/

Problem

• vision-language models
• visual token reduction
• token importance
• irreversible pruning
• grounding

Innovation

• recoverable routing
• visual token reduction
• vision-language models
• KV-cache efficiency
• training-free plug-in