BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding

2025-03-27

AI Summary

To address the challenge of long-video understanding in large video-language models (VLMs) constrained by limited context windows, this paper proposes a training-free dynamic frame selection method. First, we establish a multi-source retrieval evaluation paradigm, exposing the resource-inefficiency of uniform sampling in realistic long-video scenarios. Building on this, we design an inverse-transform sampling strategy grounded in query-frame similarity modeling, enabling inference-time dynamic focus on salient frames. Our approach achieves zero-training-cost performance gains: Video-MME accuracy improves to 56.1% (+2.3%), and MLVU to 63.4% (+4.5%), substantially outperforming baselines. The core contributions are (i) the first introduction of inverse-transform sampling for adaptive frame selection, and (ii) a novel multi-source evaluation framework tailored for long-video understanding. Together, these advances establish a new paradigm for efficient, low-cost long-video comprehension.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large video-language models (VLMs) have demonstrated promising progress in various video understanding tasks. However, their effectiveness in long-form video analysis is constrained by limited context windows. Traditional approaches, such as uniform frame sampling, often inevitably allocate resources to irrelevant content, diminishing their effectiveness in real-world scenarios. In this paper, we introduce BOLT, a method to BOost Large VLMs without additional Training through a comprehensive study of frame selection strategies. First, to enable a more realistic evaluation of VLMs in long-form video understanding, we propose a multi-source retrieval evaluation setting. Our findings reveal that uniform sampling performs poorly in noisy contexts, underscoring the importance of selecting the right frames. Second, we explore several frame selection strategies based on query-frame similarity and analyze their effectiveness at inference time. Our results show that inverse transform sampling yields the most significant performance improvement, increasing accuracy on the Video-MME benchmark from 53.8% to 56.1% and MLVU benchmark from 58.9% to 63.4%. Our code is available at https://github.com/sming256/BOLT.

Problem

• Enhancing long-form video understanding without additional training
• Addressing limited context windows in large video-language models
• Improving frame selection strategies for better video analysis

Innovation

• Query-frame similarity for frame selection
• Inverse transform sampling boosts performance
• Multi-source retrieval evaluation setting