VisualRouter: Query-Grounded Visual Sampling for Long Video Understanding

2026-07-30

AI Summary

This work addresses the challenge of long video understanding, which is hindered by the vast number of visual tokens and limited context windows in current models, making it difficult for existing sampling methods to simultaneously ensure query relevance, temporal coverage, and diversity. The authors propose VisualRouter, a training-free, plug-and-play, query-driven visual sampling framework that adaptively adjusts its strategy based on query type: it employs a hybrid relevance-coverage approach for global queries and an event-aware frame selection mechanism for local queries, thereby preserving temporal coverage while accurately aligning with query semantics. Seamlessly integrated into existing large vision-language models, VisualRouter achieves significant performance gains—5.2%, 7.7%, and 11.6% improvements on Video-MME, LongVideoBench, and MLVU respectively—when applied to Qwen2.5-VL-7B, substantially outperforming uniform sampling and other training-free baselines.

Abstract

Large vision-language models (LVLMs) have achieved significant progress in video understanding, yet understanding long videos remains challenging due to the large number of visual tokens and limited context windows. Visual sampling provides a practical solution by selecting an informative subset of frames. However, existing methods typically either rely on relevance-aware sampling, leading to redundant frame selection and insufficient temporal coverage, or adopt a fixed sampling strategy regardless of query type. In this paper, we propose VisualRouter, a training-free and plug-and-play framework for query-grounded visual sampling. VisualRouter first classifies each query as either global or local and then applies the corresponding sampling strategy. For global queries, it employs a relevance-coverage hybrid strategy that preserves temporal coverage while retaining query-relevant visual evidence. For local queries, it adopts an event-aware frame selection strategy that performs event partitioning, segment-level frame allocation, and intra-event frame selection, jointly balancing relevance, coverage, and diversity with a limited number of input frames. Experiments show that VisualRouter consistently improves multiple LVLMs over uniform sampling, achieving gains of 5.2%, 7.7%, and 11.6% on Video-MME, LongVideoBench, and MLVU with Qwen2.5-VL-7B, and outperforming existing training-free visual sampling methods under the same setting.

Problem

• long video understanding
• visual sampling
• query-grounded
• temporal coverage
• frame selection

Innovation

• query-grounded sampling
• long video understanding
• training-free framework
• event-aware selection
• visual token reduction