Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors

2026-07-17

AI Summary

This work addresses the circular dependency in long video understanding—where selecting keyframes requires prior video comprehension—by proposing DAFS, a training-free method that leverages multimodal large language models (MLLMs) to directly assess frame-query relevance without autoregressive generation. Specifically, DAFS exploits cross-modal attention patterns from specific intermediate layers during the MLLM’s verification phase to score candidate frames. It further introduces a dynamic programming-based discrete optimization strategy to jointly determine the optimal number of selected frames and the token budget per frame. Evaluated under a 32-frame budget on Video-MME, DAFS outperforms uniform sampling by up to 6.4 points and surpasses existing trainable frame selectors, demonstrating strong generalization across diverse MLLM backbones, tasks, and answer generators.

Abstract

Understanding long videos with multimodal large language models (MLLMs) requires selecting a compact set of frames from thousands of candidates, yet identifying the right frames seemingly requires understanding the video first. We resolve this circular dependency with a simple observation: cross-modal attention at validation-selected extraction layers in MLLMs already provides query-relevant frame evidence without requiring autoregressive generation. We exploit this property to build DAFS (Dynamic Attention-based Budget-aware Frame Selection), a training-free frame selector. A lightweight MLLM selector, even with only 2B parameters, can extract frame-level evidence by converting selected-layer attention into relevance scores through query-conditioned aggregation. This enables cross-frame comparison without autoregressive decoding. To handle the selector's own context constraint, we formulate the joint allocation of candidate pool size and per-frame token budget as a discrete optimization problem solved by dynamic programming. Under a 32-frame budget, our selector improves over uniform sampling by up to 6.4 points on Video-MME and outperforms prior training-based selectors under matched frame budgets, while generalizing across selector and answerer backbones, and across tasks, without retraining.

Problem

• long video understanding
• frame selection
• multimodal large language models
• test-time efficiency
• circular dependency

Innovation

• frame selection
• multimodal large language models
• attention mechanism
• training-free
• dynamic programming