Beyond Frame Selection: Rethinking Long-Video Understanding with MLLMs

2026-08-06

AI Summary

Long-form video understanding is hindered by the token-length limitations of multimodal large language models, making it challenging to simultaneously capture global narrative structure and sparse critical details. This work proposes VideoRouter, a framework that reframes long video comprehension as a multi-view collaboration problem. It constructs complementary coarse-grained (global) and fine-grained (local) evidence views through temporal hierarchical segmentation and introduces a verification-guided router to dynamically fuse these views for optimal answer selection. VideoRouter achieves the first question-agnostic coordination of multi-granularity views with adaptive reasoning, outperforming the current best frame-selection methods by 2.9 points on the VideoMME benchmark, thereby validating the efficacy of its multi-view collaborative strategy.

Abstract

Multimodal Large Language Models (MLLMs) have achieved strong progress in video understanding, yet it remains challenging because the token limitation makes MLLMs difficult to capture temporally sparse evidence. Existing methods typically rely on uniform sampling, or frame selection, but these strategies usually optimize either broad temporal coverage or local relevance, making it difficult to preserve both global storyline context and fine-grained evidence. We propose VideoRouter(VR) that rethinks long-video understanding as coordinating complementary evidence views rather than selecting a single subset of frames. It first organizes each video into a question-agnostic temporal hierarchy, which partition the video into coarse-to-fine temporally coherent segments. In this hierarchy, upper-level nodes capture broad storyline context and event progression, while lower-level nodes preserve fine-grained local details and evidence-bearing moments. This naturally gives rise to two complementary views: a global view for coverage-oriented reasoning and a local view for detail-oriented evidence recovery. We further introduce a verification-guided router to determine which view is better supported by the selected evidence and select the final answer. We validate the effectiveness of the proposed design through extensive experiments, showing that the verification-guided router effectively coordinates global and local reasoning, and that, on VideoMME, our method outperforms state-of-the-art frame selection methods by 2.9 points, respectively, under the LLaVA-Video-7B backbone. We will release the code.

Problem

• long-video understanding
• temporal sparsity
• global context
• fine-grained evidence
• token limitation

Innovation

• VideoRouter
• temporal hierarchy
• complementary views
• verification-guided routing
• long-video understanding