Unleashing Hour-Scale Video Training for Long Video-Language Understanding

2025-06-05

AI Summary

To address the performance bottleneck in long-video (≥1 hour) language understanding caused by scarce high-quality annotated data, this work introduces VideoMarathon—the first large-scale, hour-level video instruction-tuning dataset, comprising 9,700 hours of video and 3.3 million QA pairs, covering 22 short- and long-horizon multimodal understanding tasks. We propose Hour-LLaVA, a novel memory-augmented architecture featuring adaptive cache-based semantic fusion, integrated with 1-FPS sparse spatiotemporal sampling and instruction tuning to overcome long-context modeling limitations while maintaining efficient inference. This work establishes the first comprehensive training paradigm for hour-level multimodal video understanding. Evaluated on multiple long-video benchmarks, Hour-LLaVA achieves state-of-the-art performance, empirically validating both the high quality of VideoMarathon and the effectiveness of Hour-LLaVA’s architecture in long-temporal reasoning and cross-domain generalization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Video Understanding & Activity Analysis
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent long-form video-language understanding benchmarks have driven progress in video large multimodal models (Video-LMMs). However, the scarcity of well-annotated long videos has left the training of hour-long Video-LLMs underexplored. To close this gap, we present VideoMarathon, a large-scale hour-long video instruction-following dataset. This dataset includes around 9,700 hours of long videos sourced from diverse domains, ranging from 3 to 60 minutes per video. Specifically, it contains 3.3M high-quality QA pairs, spanning six fundamental topics: temporality, spatiality, object, action, scene, and event. Compared to existing video instruction datasets, VideoMarathon significantly extends training video durations up to 1 hour, and supports 22 diverse tasks requiring both short- and long-term video comprehension. Building on VideoMarathon, we propose Hour-LLaVA, a powerful and efficient Video-LMM for hour-scale video-language modeling. It enables hour-long video training and inference at 1-FPS sampling by leveraging a memory augmentation module, which adaptively integrates user question-relevant and spatiotemporal-informative semantics from a cached full video context. In our experiments, Hour-LLaVA achieves the best performance on multiple long video-language benchmarks, demonstrating the high quality of the VideoMarathon dataset and the superiority of the Hour-LLaVA model.

Problem

• Addressing scarcity of annotated long videos for training
• Enabling hour-long video understanding with large dataset
• Improving long video-language model performance efficiently

Innovation

• Large-scale hour-long video dataset VideoMarathon
• Memory augmentation module for hour-scale training
• Hour-LLaVA model for long video understanding