Calibrate Before Reason: Robust Visual Token Reduction against Semantic Drift in VLMs

2026-07-30

AI Summary

This work addresses the high computational overhead of vision-language models caused by lengthy visual token sequences, a challenge exacerbated by existing compression methods that often induce semantic drift and degrade comprehension accuracy. To overcome this, the authors propose CaRe, a training-free framework that introduces a novel “calibrate-then-reason” paradigm. Prior to inference, CaRe semantically calibrates compressed representations through perturbation-robust calibration anchors and a confidence-gated mechanism. By integrating multi-directional perturbation stability analysis with reliable signal injection from unselected tokens, CaRe effectively mitigates semantic drift without requiring retraining. Experiments demonstrate that CaRe retains 96.4% of the original model performance while preserving only 5.6% of visual tokens (94.4% pruning rate), achieving a 2.30× end-to-end inference speedup and significantly outperforming current compression approaches.

Abstract

Large Vision-Language Models (VLMs) suffer from prohibitive inference overhead due to long sequences of visual tokens. However, existing visual token reduction methods mainly improve efficiency by pruning or compressing redundant tokens without examining whether the resulting representation remains semantically consistent with the original representation. Mapping the original N-token visual sequence to K tokens may discard, dilute, or misassign critical visual cues, triggering severe semantic drift that deviates the VLM's understanding. In this paper, we first introduce the principle of 'Calibrate Before Reason' to visual token reduction and propose CaRe, a training-free robust framework that calibrates compact visual representations before reasoning to preserve semantic fidelity in VLMs. CaRe consists of two mutually complementary modules: 1) Perturbation-Robust Calibration Anchoring, which identifies calibration anchors with stable model-side influence under multi-directional perturbations; 2) Confidence-Gated Token Calibration, which extracts reliable calibration signals from unselected tokens and injects them into anchors. Extensive evaluations across diverse VLM architectures and benchmarks verify that CaRe outperforms state-of-the-art token reduction baselines. While pruning 94.4% of visual tokens, our method retains 96.4% of the original full-token performance, delivering up to 2.30 times faster end-to-end inference speed relative to unpruned vanilla models.

Problem

• visual token reduction
• semantic drift
• vision-language models
• semantic consistency
• inference efficiency

Innovation

• visual token reduction
• semantic drift
• calibration
• vision-language models
• training-free