MaxInfo: A Training-Free Key-Frame Selection Method Using Maximum Volume for Enhanced Video Understanding

2025-02-05

AI Summary

To address the critical issue of key information loss and severe redundancy caused by uniform frame sampling in Video Large Language Models (VLLMs), this paper proposes a training-free, plug-and-play keyframe selection method. The core innovation lies in the first application of the Maximum Volume (MaxVol) principle to zero-shot video frame selection: frames are selected from the embedding space of a frozen pre-trained vision encoder such that their induced simplex attains maximal volume—thereby jointly optimizing information coverage and diversity while suppressing redundancy. The method requires no fine-tuning, external annotations, or architectural modifications. Extensive experiments on benchmarks including LongVideoBench and EgoSchema demonstrate consistent and significant performance gains (+3.28% to +6.4%) for state-of-the-art VLLMs (e.g., LLaVA-Video-7B/72B), substantially outperforming uniform sampling while maintaining full compatibility with existing VLLM frameworks.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Modern Video Large Language Models (VLLMs) often rely on uniform frame sampling for video understanding, but this approach frequently fails to capture critical information due to frame redundancy and variations in video content. We propose MaxInfo, a training-free method based on the maximum volume principle, which selects and retains the most representative frames from the input video. By maximizing the geometric volume formed by selected embeddings, MaxInfo ensures that the chosen frames cover the most informative regions of the embedding space, effectively reducing redundancy while preserving diversity. This method enhances the quality of input representations and improves long video comprehension performance across benchmarks. For instance, MaxInfo achieves a 3.28% improvement on LongVideoBench and a 6.4% improvement on EgoSchema for LLaVA-Video-7B. It also achieves a 3.47% improvement for LLaVA-Video-72B. The approach is simple to implement and works with existing VLLMs without the need for additional training, making it a practical and effective alternative to traditional uniform sampling methods.

Problem

• Enhances video understanding by selecting key frames
• Reduces frame redundancy and preserves content diversity
• Improves performance of Video Large Language Models

Innovation

• Training-free key-frame selection method
• Maximum volume principle application
• Enhances video understanding performance