PCA: Persistence-Aware Compression and Aggregation for Fast Video Large Language Models

2026-07-22

AI Summary

This work addresses the inefficiency of video large language models (VLLMs) caused by redundant frames in long videos. The authors propose a training-free, persistence-aware compression and aggregation method that dynamically downsamples redundant frames prior to encoding while enhancing key frames with temporal context from neighboring frames to preserve high-fidelity visual information. Introducing, for the first time, a persistence-aware mechanism into frame compression, the approach integrates dynamic downsampling (DD) with a persistence-aware motion enhancement (PAME) module to achieve efficient yet information-preserving video understanding. The method is model-agnostic and compatible with any VLLM, delivering 1.8–2.5× inference speedup over baselines while maintaining or even improving accuracy, significantly outperforming existing techniques.

Abstract

Despite advances in Video Large Language Models (VLLMs) that have displayed promising outcomes in video understanding, the redundancy in the long-duration frames remains a hindrance to efficient reasoning. This paper introduces a training-free $\mathbf{P}$ersistence-Aware $\mathbf{C}$ompression and $\mathbf{A}$ggregation (PCA) method designed to preserve high-fidelity raw visual information before the encoding stage. PCA can be built on arbitrary VLLMs and consists of two modules: 1) A Dynamic Downsampling (DD) module that adaptively removes redundant frames by analyzing frame-wise similarity. 2) A Persistence-Aware Motion Enhancement (PAME) module that enriches each selected keyframe by aggregating the temporal context of its neighbors, ensuring that essential information is preserved even after aggressive frame reduction. Our approach substantially reduces the computation of long-context modeling, while enhancing the performance of the baseline model. Extensive experiments demonstrate that PCA consistently outperforms existing state-of-the-art approaches in both efficiency and accuracy, achieving a speedup of 1.8$\times$ to 2.5$\times$ compared to the baseline VLLM. The code is open-sourced at https://github.com/Heisenberg10110/PCA.

Problem

• Video Large Language Models
• frame redundancy
• long-duration video
• efficient reasoning
• temporal redundancy

Innovation

• Persistence-Aware Compression
• Dynamic Downsampling
• Temporal Context Aggregation
• Training-Free Acceleration
• Video Large Language Models