When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding

2026-08-04

AI Summary

This work addresses the challenge of efficiently selecting a minimal set of keyframes from long videos as visual evidence while preserving inference accuracy. The authors propose EcoFrame, a novel framework that introduces the first training-free, adaptive evidence scheduling mechanism. EcoFrame leverages feedback from off-the-shelf vision-language models to dynamically allocate frame-sampling budgets via entropy gating and employs attention guidance to prioritize information-dense regions, thereby maintaining global coverage while substantially reducing computational overhead. Experiments demonstrate that EcoFrame outperforms existing methods across multiple benchmarks, achieving an average accuracy of 64.4% with Qwen2.5-VL—0.9 percentage points higher than BOLT—and delivering up to 13.5× faster inference.

Abstract

Efficient long-video understanding requires vision--language models (VLMs) to reason over a small number of frames selected as sparse visual evidence. Existing relevance-based methods rely on static one-shot selection with fixed frame budgets and candidate pools, while agent-based schedulers achieve adaptivity through costly multi-round reasoning and interactive search. We propose EcoFrame, a training-free framework for low-overhead query-adaptive visual evidence scheduling. EcoFrame leverages the VLM's inference feedback to determine when to increase the frame budget and where to search for additional candidate evidence. Specifically, entropy-gated budget scheduling uses output uncertainty to stop early when the current evidence is sufficient or progressively expand the frame budget otherwise. Meanwhile, attention-guided candidate proposal converts frame-level attention into a temporal prior, enabling dense local search in informative regions while preserving global coverage when attention is diffuse. Experiments on Video-MME, LongVideoBench, and MLVU demonstrate that EcoFrame achieves a better accuracy--efficiency trade-off across multiple VLM backbones. On Qwen2.5-VL, EcoFrame achieves an average accuracy of 64.4, surpassing BOLT at 63.5, while providing a $1.85\times$ speedup over AKS and BOLT. Compared with the agent-based A.I.R., EcoFrame maintains comparable accuracy with up to a $13.5\times$ inference speedup. Code will be available at https://github.com/AK-DREAM/EcoFrame.

Problem

• long video understanding
• visual evidence scheduling
• frame selection
• vision-language models
• efficiency-accuracy trade-off

Innovation

• adaptive scheduling
• visual evidence selection
• entropy-gated budgeting
• attention-guided search
• training-free framework