Coverage-Driven Adaptive Keyframe Selection for Video Understanding

2026-08-01

AI Summary

Long-form video understanding faces high computational costs and limitations in existing keyframe selection methods, which struggle to adaptively adjust sampling density and coverage based on query content. This work proposes CSES, the first approach to formalize keyframe selection as a coverage optimization problem. Without requiring training, CSES estimates the saliency of frame-query relevance distributions to guide active sampling and constructs a monotone submodular coverage function incorporating semantic relevance, temporal structure, and visual redundancy. A greedy algorithm ensures near-optimal solutions with theoretical guarantees. Evaluated across two benchmarks using four large vision-language models, CSES reduces the number of scored frames by 4–13× and input keyframes by 18.4%–20.5% compared to baselines, while maintaining accuracy and achieving 3.1–5.4× faster frame selection.

Abstract

Recent advances in large vision-language models (LVLMs) have enabled long-video understanding and analysis. However, processing the large number of frames in a video incurs substantial computational overhead. Existing methods reduce LVLM inference costs by scoring frame-query relevance before inference and selecting keyframes accordingly. Nevertheless, the distribution of relevant frames varies across queries, and these methods often need to score hundreds or thousands of frames. To address this limitation, we propose CSES, a training-free semantic keyframe selector that adaptively determines the numbers of frames to score and keyframes to select. CSES estimates the prominence of the frame-query relevance profile to guide active acquisition and adapt the temporal coverage of each input. It then formulates keyframe selection as a coverage problem that jointly accounts for semantic relevance, temporal redundancy, and visual redundancy. Active acquisition and keyframe selection terminate based on coverage saturation. The selection objective is monotone and submodular, enabling greedy optimization with a standard approximation guarantee. Experiments with four LVLMs on two benchmarks show that our method preserves accuracy while scoring $4$-$13\times$ fewer frames and selecting $18.4\%$-$20.5\%$ fewer input keyframes than existing baselines. CSES further achieves a $3.1$-$5.4\times$ speedup in frame selection over baselines.

Problem

• video understanding
• keyframe selection
• computational overhead
• frame-query relevance
• large vision-language models

Innovation

• adaptive keyframe selection
• coverage-driven
• submodular optimization
• training-free
• video understanding