FORGE: Frame Orthogonality in Relevance Geometry for Long-Form Video Understanding

2026-07-28

AI Summary

This work addresses the challenge of degraded performance in multimodal large language models (MLLMs) when processing long videos, where relevant frames are sparse and obscured by irrelevant content. The authors propose a training-free, model-agnostic keyframe selection method that operates at inference time. By modeling query relevance as an orthogonal direction in a pretrained multimodal embedding space, the approach geometrically optimizes both relevance and diversity of selected frames. A novel algorithm is introduced to select the subset of frames maximizing information coverage. Evaluated on Video-MME and LongVideoBench, the method substantially outperforms existing training-free baselines, achieving up to a 2× improvement in keyframe recall and boosting question-answering accuracy by as much as 8.7 percentage points across eight open-source MLLMs.

Abstract

Multimodal large language models (MLLMs) have enabled long-form video understanding at a scale that was not previously possible. However, the density of relevant content decreases sharply as video sequence length increases, and exposing the model to more irrelevant content measurably reduces its accuracy. In this paper, we address the problem of maximizing query-relevant information in a frame subset selected at inference time, without training. FORGE (Frame Orthogonality in Relevance Geometry) is a model-agnostic method that induces a query-conditioned geometry on a pretrained multimodal embedding space, unifying relevance and diversity into a single objective. In this space, frames that cover independent query-relevant directions are far apart, and selecting the subset of maximum information captures diverse query-relevant content within the budget. Experiments on Video-MME and LongVideoBench at budgets of 16, 32, and 64 frames show that FORGE improves the unified keyframe selection score by 11.0-15.3 points over the strongest training-free baseline and up to doubles keyframe recall (0.415 vs. 0.204 at K=64 on Video-MME). The gains extend to question answering, where accuracy improves in every evaluated setting across eight open-source MLLMs spanning 4B to 32B parameters, by up to 8.7 points over uniform sampling and 5.2 points over the strongest baseline. Our findings suggest that aligning the embedding space with the query's high-dimensional structure is a promising direction for inference-time video understanding.

Problem

• long-form video understanding
• query-relevant information
• keyframe selection
• multimodal large language models
• irrelevant content

Innovation

• Frame Selection
• Relevance Geometry
• Model-Agnostic
• Multimodal Embedding
• Long-Form Video Understanding