FORTE: Adaptive Scoring and Exact Keyframe Selection for Long-Video Question Answering

2026-09-30

AI Summary

This study addresses the limitation of fixed candidate pools in long-form video question answering, which often overlook critical frames. To this end, we propose a training-free keyframe selection framework that leverages Gaussian process regression to achieve adaptive scoring with linear complexity. Furthermore, it employs an exact optimal subset algorithm grounded in a logarithmic coverage structure for global combinatorial optimization, effectively balancing semantic relevance with temporal coverage. Extensive experiments across four benchmarks demonstrate that the proposed framework achieves state-of-the-art accuracy. Notably, it exhibits strong generalization and robustness when integrated with diverse scoring mechanisms and various multimodal large language models.

Abstract

Query-aware keyframe selection enables multimodal large language models (MLLMs) to process long videos using only a small set of question-relevant frames. Existing score-based methods, however, typically search within a fixed, uniformly sampled candidate pool, preventing evidence outside this pool from ever being selected. Given a limited relevance-scoring budget, the key challenge is to allocate evaluations adaptively to promising frames while continuing to explore underrepresented temporal regions. We introduce FORTE, a training-free framework that addresses this challenge through two stages: adaptive relevance scoring and global keyframe optimization. Starting from sparse, uniformly distributed observations, our efficient Gaussian-process relevance predictor estimates relevance for unscored frames, exploiting temporal locality and the approximately banded kernel structure to reduce the core computation from cubic to linear time in the number of frames for fixed bandwidth. The scoring stage then selects which frames to score next by balancing predicted relevance with temporal coverage, prioritizing promising regions while also exploring less-represented parts of the video. The optimization stage selects the final keyframes by maximizing an objective that jointly captures measured relevance and temporal coverage. We derive an exact algorithm that leverages the logarithmic coverage structure to identify the optimal subset of the scored candidate pool in time linear in the pool size, for a fixed final-frame budget. Experiments on four long-video question-answering benchmarks show that FORTE achieves the highest observed mean accuracy among the compared selectors under every tested scoring budget. Further evaluations demonstrate its consistent effectiveness across different relevance scorers and downstream MLLMs.

Problem

• Long-video question answering
• Keyframe selection
• Adaptive scoring
• Relevance budget
• Multimodal large language models

Innovation

• Adaptive Scoring
• Exact Keyframe Selection
• Gaussian Process
• Long-Video QA
• Training-free Framework