CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models

2026-08-02

AI Summary

This work addresses the substantial computational and memory overhead in the prefilling stage of vision-language models for video understanding, which arises from processing a large number of visual tokens. Existing compression methods struggle to balance efficiency with content adaptivity. To overcome this, the authors propose a decoupled, recursive adaptive compression framework that selects tokens for fusion based on parameter-free global similarity, followed by a linear fusion strategy incorporating position-aware weighting and channel-wise adaptive gating. This approach is query-agnostic, preserves spatiotemporal coordinates, and requires no additional alignment training, thereby maintaining compatibility with the input distribution of pretrained language models. At approximately 8× compression, it retains 97% of the backbone model’s average accuracy and significantly outperforms current state-of-the-art methods across multiple video benchmarks.

Abstract

In video understanding, vision-language models (VLMs) must ingest massive numbers of visual tokens, causing the computational and memory cost of the prefill stage to rise sharply. Such visual sequences are highly redundant along the spatio-temporal dimension, yet a high compression ratio is often accompanied by the loss of critical details. Existing token-compression methods either employ heuristic, training-free compression with limited content adaptivity or introduce additional modules that require expensive alignment training, leaving the trade-off between efficiency and adaptivity unresolved. To alleviate this limitation, we propose CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens. CRAFT recursively merges tokens by decoupling parameter-free token selection from learnable token fusion: global similarity determines which tokens to merge, while a position-aware weighting module and a content-adaptive channel-wise gate learn how to fuse them. The whole compression pipeline is query-agnostic. Because every retained token is a linear combination of the original tokens, CRAFT preserves their true spatio-temporal coordinates and stays aligned with the pre-trained language model's input distribution. Experiments on multiple representative video benchmarks show that CRAFT consistently outperforms prior state-of-the-art token-compression methods. At about $8\times$ compression, it retains roughly $97\%$ of the backbone's average accuracy and shows significant efficiency improvement.

Problem

• video understanding
• vision-language models
• token compression
• spatio-temporal redundancy
• efficiency-adaptivity trade-off

Innovation

• token compression
• vision-language models
• recursive fusion
• content-adaptive gating
• spatio-temporal redundancy