LENS: Adaptive Spatio-Temporal Zooming for Keyframe Sampling in Long-Form Videos

2026-07-27

AI Summary

This work addresses the challenge of long video understanding, which is constrained by the limited context length of multimodal large language models and the inability of existing keyframe sampling methods to simultaneously preserve spatial detail and temporal coverage. The authors propose a training-free, query-driven adaptive sampling framework that introduces, for the first time, a dynamic spatiotemporal zooming mechanism guided by textual queries. This mechanism “zooms in” to focus on relevant regions within individual frames for fine-grained detail capture while “zooming out” to expand the temporal window for aggregating long-range contextual information, thereby balancing local and global cues under a fixed frame budget. The method seamlessly integrates with existing vision-language models and achieves substantial performance gains across multiple long-video benchmarks, notably improving Qwen2.5-VL’s accuracy on Video-MME from 53.3% to 60.7%.

Abstract

Despite rapid progress in Multi-modal Large Language Models (MLLMs), understanding long-form videos is still bottlenecked by limited context windows. While recent keyframe sampling methods attempt to mitigate this by distilling video inputs into a compact set of query-relevant frames, navigating the vast spatio-temporal search space remains challenging, as spatial detail and temporal coverage often conflict. To address this, we introduce LENS, a training-free keyframe sampling framework that dynamically decides when to zoom in for fine-grained details and when to zoom out for broader context based on the text query. Concretely, LENS adaptively allocates a limited frame budget between spatial zoom-ins, which highlight query-relevant regions within individual frames, and temporal zoom-outs, which expand the temporal scope through multi-frame aggregation, enabling the model to reason across multiple granularities while capturing both high-fidelity details and long-range context. Across diverse long-form video benchmarks, LENS consistently outperforms prior state-of-the-art keyframe sampling methods and delivers substantial gains over uniform sampling, improving Video-MME accuracy from 53.3% to 60.7% with Qwen2.5-VL.Code is available at https://github.com/zhangce01/LENS.

Problem

• long-form videos
• keyframe sampling
• spatio-temporal
• context window
• multi-modal understanding

Innovation

• adaptive spatio-temporal zooming
• keyframe sampling
• training-free framework
• multi-granularity reasoning
• long-form video understanding