Think in Sets for Streaming Video Token Compression

2026-08-02

AI Summary

This work addresses the high prefill latency and substantial memory overhead in streaming video large language models caused by the continuous accumulation of visual tokens, a challenge exacerbated by existing training-free compression methods that overlook marginal gain interactions among tokens. The authors propose NovaCov, the first training-free, plug-and-play, set-based token compression method tailored for streaming video. Framing compression as a set selection problem, NovaCov employs a capacity-constrained, recency-weighted historical reference bank to optimize a dual-branch submodular coverage objective—based on facility location functions—during causal per-frame pruning, balancing frame representativeness with coverage of historically underrepresented information. With theoretical approximation guarantees, NovaCov outperforms existing training-free approaches on both streaming and offline benchmarks, retaining only a small fraction of tokens while preserving 99.6% of ReKV accuracy and reducing LLM prefill latency by 46%.

Abstract

Streaming VideoLLMs process frames causally while visual tokens grow continuously, making compression essential for controlling prefilling latency and memory. Existing training-free methods independently rank tokens, ignoring marginal-gain interactions among retained tokens. We argue that streaming video token compression should instead be formulated as set selection, where each candidate is valued by what it adds beyond the tokens already retained. Unlike existing set-wise methods designed for offline tasks, streaming makes causal, frame-by-frame pruning decisions, so modeling cross-frame interactions requires an explicit historical reference. This creates a reference-set dilemma: the reference must adequately represent previously conveyed content while remaining bounded for real-time inference. We introduce NovaCov, to our knowledge the first training-free, plug-and-play set-wise token compressor designed for streaming video. NovaCov maintains a capacity-bounded, recency-weighted Historical Reference Bank and optimizes a dual-branch submodular coverage objective that preserves representative current-frame content while prioritizing information insufficiently covered by history. Both branches are facility-location functions, so greedy selection retains the classical (1-1/e) approximation guarantee. Across streaming and offline benchmarks, NovaCov outperforms existing training-free compression methods, retaining 99.6% of ReKV accuracy while reducing LLM prefilling latency by 46%.

Problem

• streaming video
• token compression
• set selection
• prefilling latency
• historical reference

Innovation

• set-wise token compression
• streaming video LLMs
• submodular optimization
• historical reference bank
• training-free compression