AdaCodec: A Predictive Visual Code for Video MLLMs

2026-06-01

AI Summary

This work addresses the inefficiency of existing video multimodal large language models (MLLMs), which encode each frame independently, leading to redundant visual tokens and poor exploitation of temporal consistency. The authors introduce predictive coding into the visual input interface of video MLLMs for the first time, proposing an adaptive predictive visual encoding mechanism. This approach transmits a full reference frame only when the scene is difficult to predict; otherwise, it represents inter-frame changes using compact P-tokens. By integrating conditional prediction cost estimation, motion and residual encoding, and compressed P-token representations, the method dynamically allocates visual tokens under a fixed budget. Evaluated against Qwen3-VL-8B, the proposed model achieves superior performance across all metrics using the same token budget, matches or exceeds its accuracy with only one-seventh the tokens on long-video tasks, and reduces first-token latency from 9.26s to 1.62s.

Abstract

Video is temporally redundant: adjacent frames usually share most objects, background, and layout. Yet existing video multimodal large language models (video MLLMs) usually encode each sampled frame as an independent RGB image, causing visual tokens to repeat content already present in earlier frames. This suggests a more direct video interface: send a full reference frame only when the scene cannot be predicted well from prior context, and otherwise transmit a compact description of inter-frame changes. We call this interface a \emph{predictive visual code}, and instantiate it for video MLLMs as \textbf{AdaCodec}. AdaCodec spends full visual tokens on a reference frame only when its conditional predictive cost is high; otherwise, it encodes inter-frame changes, including motion and prediction residuals, as compact P-tokens. Across all eleven benchmarks, AdaCodec improves over the Qwen3-VL-8B per-frame RGB baseline at a matched visual-token budget. Even at $1/7$ the budget, AdaCodec with 32k tokens surpasses the 224k baseline on all long-video benchmarks; on five general-video benchmarks, it raises the average score while substantially cutting time-to-first-token from 9.26s to 1.62s.

Problem

• video MLLMs
• temporal redundancy
• visual tokens
• frame encoding
• predictive coding

Innovation

• predictive visual code
• AdaCodec
• video MLLMs
• temporal redundancy
• P-tokens