Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding

2026-08-06

AI Summary

This work addresses the limitations of existing long video understanding methods, which rely on external scorers and fixed heuristics that fail to align with the internal evidence of multimodal large language models (MLLMs) and cannot adapt to the non-uniform spatiotemporal information density in videos. To overcome these challenges, the authors propose EviSelect, a novel framework that leverages the target MLLM’s internal attention as visual evidence to construct a lightweight, dynamic selector. By employing sparse pre-filling to establish structured priors, EviSelect enables distribution-aware dynamic sampling that adaptively adjusts temporal localization, local sampling rates, and spatial resolution. The method further combines compressed visual inputs with sparse attention to approximate full attention maps and optimizes the stochastic sampling strategy via GRPO. Evaluated on three benchmarks, EviSelect achieves state-of-the-art performance while reducing visual tokens by approximately 50% and accelerating end-to-end inference by 3.9×.

Abstract

Recent advancements in MLLM-based long-form video understanding have mitigated inference-time computational cost and limited context lengths by selecting query-relevant frames. However, existing approaches predominantly rely on external proxy scorers and rigid heuristic rules, inevitably suffering from misalignment with the target MLLM's intrinsic evidence and failing to accommodate the non-uniform spatiotemporal information density. In this paper, we propose a fine-grained dynamic visual selection framework named EviSelect, grounded in the target MLLM internal attention evidence. Our method efficiently probes visual evidence via sparse prefilling as a structured prior to guide distribution-aware dynamic sampling. Specifically, we efficiently approximate attention maps of the target MLLM using highly compressed visual inputs and sparse attention, well-aligned to the full counterpart. Conditioned on three complementary attention components derived from this prior, we design a lightweight selector that not only precisely locates query-relevant timestamps but also adaptively adjusts the local sampling rate and spatial resolution. To enable evidence-conditioned spatiotemporal sampling, we formulate the selector as a stochastic policy and optimize it via GRPO under a joint accuracy--efficiency reward. By rewarding correct predictions under lower visual cost through group-relative comparisons, our method encourages the policy to allocate computation dynamically according to the information density of each video. Across three long video understanding benchmarks, EviSelect achieves superior performance compared to existing methods while reducing selected visual tokens by about 50\% and achieving a 3.9x end-to-end speedup.

Problem

• long video understanding
• frame selection
• evidence alignment
• spatiotemporal information density
• MLLM

Innovation

• evidence-driven selection
• dynamic visual sampling
• MLLM attention evidence
• spatiotemporal token reduction
• GRPO optimization