Advancing Video-Text Pretraining with Multi-View Captions

2026-09-28

AI Summary

This study addresses the limitations of sparse web descriptions, insufficient spatiotemporal semantics, and high noise in direct generation for video-text pre-training. To overcome these challenges, we propose a multi-view captioning and granularity-aware representation framework based on multimodal large language models. Specifically, we construct a complementary captioning system encompassing summaries, details, reasoning refinements, and semantic positive samples to synthesize high-quality supervision from tens of millions of videos. Furthermore, a multi-granularity independent CLS token mechanism is designed to optimize textual representations. Experimental results demonstrate that our approach significantly improves both zero-shot and fine-tuned performance across multiple retrieval benchmarks, achieving superior video-text understanding capabilities with a comparatively smaller corpus size.

Abstract

Video-text pretraining has achieved remarkable progress through the scaling of models and datasets, yet the quality of language supervision remains underexplored. Existing web-scale datasets often provide only a single sparse caption per video that fails to capture rich spatiotemporal semantics, while directly using captioning models can generate noisy descriptions. We propose a large-scale multimodal large language model-based supervision generation framework that improves supervision diversity, fidelity, and semantic coverage. Starting from 10 million videos, our approach generates multi-view captions (MVC) through complementary summary and detailed captions, reasoning-based refinement, and semantic positive caption generation. To effectively exploit supervision at different granularities, we further introduce a granularity-aware text representation with separate CLS tokens for summary and detailed views. We pretrain video-text models using the resulting supervision corpus and evaluate them across standard, fine-grained and detailed text-to-video retrieval benchmarks. Our approach consistently improves both zero-shot and fine-tuned performance while using smaller pretraining corpora than existing methods, demonstrating the importance of rich and complementary textual supervision for video-text pretraining. Project page: https://rvandeghen.github.io/mvc/

Problem

• Video-text pretraining
• Language supervision quality
• Multi-view captions
• Spatiotemporal semantics
• Noisy descriptions

Innovation

• Video-Text Pretraining
• Multi-View Captions
• Multimodal Large Language Model
• Granularity-Aware Representation
• Text-to-Video Retrieval