Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression

2026-08-03

AI Summary

Existing visual-language model compression methods rely solely on importance scores of individual visual tokens, overlooking the fact that textual queries actually depend on collective attention over the entire visual input, often leading to the loss of crucial complementary information. This work proposes GMC, a training-free compression approach that, for the first time, decouples compression into two intertwined components: message carrier selection and group message reconstruction. By leveraging query anchoring, appearance- and coordinate-aware evidence allocation, and a state transition mechanism, GMC migrates information from discarded tokens to retained representatives prior to physical compression, thereby constructing a compact coreset that faithfully preserves decoder-relevant messages. Evaluated on Qwen2.5-VL-7B, GMC-H2 retains only 19.8% of visual tokens yet recovers 97.78% of full-model performance, while GMC-L16 even achieves 100.36%, demonstrating consistent effectiveness across multiple vision-language models and benchmarks.

Abstract

Modern vision language models (VLMs) turn high-resolution images into long sequences of visual tokens. Every token traverses the language decoder and persists in its prompt KV cache, inflating inference cost and motivating aggressive visual compression. Existing score-based methods assign each token an independent importance score and retain the Top-K. However, text queries consume collective, signed attention messages from the visual population, not isolated patches. Consequently, equally sized Top-K sets can repeatedly cover one salient region, omit sparse but complementary evidence and discard information carried by the removed population. We therefore formulate faithful visual compression as constructing a compact coreset for decoder messages, and introduce our training-free Grounded Message Coreset Pruning (GMC) which jointly allocates support across query-grounded, appearance, and coordinate-aware evidence, then transports discarded states into selected representatives at their original multimodal positions before physical compaction and native attention resume. This decomposes faithful compression into two coupled components, including selecting carriers that cover the required message modes and realizing the signed population message on those carriers. We further derive bounds connecting their errors to signed-message distortion, visual innovation, and candidate-margin stability. Experiments across multiple VLM families and diverse benchmarks demonstrate strong performance, with GMC-H2 retaining 97.78% Full-relative mean capability on Qwen2.5-VL-7B using 80.2% fewer visual tokens, while GMC-L16 reaches 100.36%. Controlled interventions verify that collective support and population realization jointly drive these gains.

Problem

• vision-language models
• visual compression
• attention messages
• coreset
• token pruning

Innovation

• Grounded Message Coreset
• VLM Compression
• Signed Attention Messages
• Training-free Pruning
• Multimodal Coreset