Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention

2026-03-23

AI Summary

This work addresses the high computational cost of video large language models caused by the excessive number of visual tokens, a challenge exacerbated by existing compression methods that suffer from spatiotemporal imbalance and critical information loss under extremely low token retention rates. The authors propose a unified spatiotemporal token compression framework that reformulates compression as an allocation task within a global retention pool. By jointly leveraging attention weights and semantic similarity, the method selects high-contribution, low-redundancy tokens globally, while clustering and reintegrating unselected tokens to preserve contextual integrity. Furthermore, a text-aware merging mechanism inside the LLM enables secondary compression. This approach discards the conventional assumption of separable spatial and temporal compression, operates without retraining, and is plug-and-play. Retaining only ~2% of visual tokens maintains 90.1% of baseline performance while reducing FLOPs to 2.6%, substantially lowering inference latency and memory consumption with strong generalization capability.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Video large language models (Video-LLMs) face high computational costs due to large volumes of visual tokens. Existing token compression methods typically adopt a two-stage spatiotemporal compression strategy, relying on stage-specific metrics and an implicit assumption of spatiotemporal separability. Under extremely low retention ratios, however, such approaches often result in unbalanced allocation and loss of visual evidence essential for question answering. We reformulate token compression as a spatiotemporal allocation task within a global token retention pool. We propose a unified selection mechanism that integrates attention weights and semantic similarity to globally select tokens with high contribution and low redundancy. Unselected tokens are merged via clustering and refilled, preserving information integrity. Inside the LLM, we further introduce text-aware merging to perform secondary compression based on query relevance. Without requiring retraining, our method serves as a plug-and-play module compatible with existing Video-LLMs. Experiments show that retaining only about 2% of visual tokens preserves 90.1% of baseline performance across multiple benchmarks, while reducing FLOPs to roughly 2.6%. These benefits generalize across diverse backbones, decreasing end-to-end inference latency and memory consumption. Our unified spatiotemporal token compression strategy establishes the state-of-the-art in video understanding under ultra-low token retention.

Problem

• Video-LLMs
• token compression
• spatiotemporal allocation
• ultra-low retention
• visual tokens

Innovation

• spatiotemporal token compression
• video-LLMs
• ultra-low retention
• unified selection mechanism
• text-aware merging