video dataset collection

This work proposes a systematic data engineering methodology to address the engineering challenges of processing and managing raw video data for large-scale video foundation model training. By leveraging metadata-driven data cleaning, multi-stage filtering, and an inference-aware architecture—combined with the Lavender Data management system, μP hyperparameter transfer, and hyperspherical geometric constraint optimization—the authors efficiently construct a high-quality training set from approximately 50 million video clips. Using this curated dataset, they successfully train Summer-22B, a 22-billion-parameter video foundation model. This study provides the first empirical validation that systematic data engineering plays a dominant role in enhancing model performance, underscoring the critical importance of data quality and structural integrity in large-scale video modeling.

To address the high cost and low efficiency of manual annotation in video object detection and segmentation, this paper proposes a lightweight, end-to-end automated annotation framework. Methodologically, it introduces the first deep integration of YOLOv8-based video tracking and SAM-based interactive segmentation, augmented by an adaptive frame-sampling strategy and a Gradio-powered visual interface, resulting in a modular, open-source, and extensible prototype system. Key contributions include: (i) a lightweight co-design of tracking and segmentation models; (ii) efficient, interactive annotation support for multi-object, long-duration videos; and (iii) rapid generation of annotated datasets enabling a closed-loop annotation–training pipeline. Experiments on multiple public video benchmarks demonstrate over 10× higher annotation throughput compared to manual labeling, strong inter-annotator consistency, and substantial performance gains for downstream detection and segmentation models trained on the generated data.

This work addresses the scarcity of large-scale, publicly available, and reproducible benchmark datasets in multimodal recommendation research. To this end, we present the first end-to-end traceable data processing pipeline built upon MovieLens-10M and MovieLens-20M, enriching them with multimodal signals—including plot summaries, movie posters, and trailer videos—and extracting textual, visual, audio, and video features using state-of-the-art encoders. The resulting datasets, M3L-10M and M3L-20M, constitute two large-scale multimodal benchmarks for movie recommendation. We provide complete mappings to original data sources, detailed feature extraction protocols, and releases in multiple formats to significantly enhance reproducibility. Both qualitative and quantitative analyses validate the effectiveness and utility of the proposed datasets, thereby advancing research in multimodal recommender systems.

Existing video datasets suffer from coarse-grained temporal segmentation, unstructured text descriptions, and the absence of rigorous quality assessment—limiting fine-grained alignment in controllable video generation. To address these challenges, Koala-36M introduces a systematic solution: (1) constructing a large-scale dataset comprising 36 million high-quality video clips; (2) proposing the Video Training Suitability Score (VTSS), a novel multi-metric quality evaluation framework; (3) developing a high-accuracy transition detection method based on linear classification over probability distributions; and (4) generating structured, long-form textual descriptions averaging 200 words per clip, coupled with optimized fine-grained conditional embeddings. Experiments demonstrate substantial improvements in spatiotemporal text-video alignment accuracy and generation consistency. The codebase, dataset, and full preprocessing pipeline are publicly released.

This work addresses key challenges in egocentric video analysis: temporal overlap of audio events, misalignment between audio and visual modalities in both timing and semantics, and ambiguity in audio-only annotations. To this end, we introduce EPIC-SOUNDS—the first fine-grained audio event dataset explicitly designed for causal action understanding. Built upon 100 hours of kitchen audio from EPIC-KITCHENS-100, it comprises 78.4k classifiable and 39.2k unclassifiable sound events, covering 44 audio-discriminable actions. We propose a novel “object-material pair + visual verification” paradigm to establish the first physics-grounded sound-action mapping benchmark. Annotation employs a joint strategy combining crowdsourced temporal labeling, free-text description clustering, and cross-modal ambiguity filtering. Evaluation on SOTA models (e.g., AudioCLIP, PANNs) reveals substantial limitations in recognizing “audible actions.” The dataset and baseline code are publicly released to advance audio-driven action understanding and embodied perception research.

Existing Foley sound datasets generally suffer from insufficient quality and coarse annotations, hindering data-driven research in classification, retrieval, and synthesis. To address this gap, this work introduces and publicly releases FoleySet—a large-scale Foley dataset comprising 10,000 audio clips meticulously recorded following professional Foley practices. The dataset features a two-tier manual semantic annotation scheme that precisely aligns synchronous sound effects with on-screen human actions, such as footsteps, clothing rustles, and prop manipulations. FoleySet is the first to offer multi-level annotations, standardized formatting, and a permissive Creative Commons license, thereby filling a critical resource void in the field. It provides strong support for Foley-related audio tasks and advances research toward automated audiovisual content production.

This work addresses the limitations of existing audio-visual synchronization evaluation methods, which struggle to disentangle temporal alignment from semantic consistency and suffer from coupling biases in data construction. We propose the first structured and scalable benchmark framework that enables independent assessment of temporal synchronization and semantic correspondence. Through a hybrid pipeline combining automated filtering and human verification, we construct a large-scale dataset comprising 3,269 videos and 38,390 samples across three audio categories—speech, music, and environmental sounds—and ten diverse scenarios. The dataset ensures authentic on-screen sound sources and supports both multimodal alignment analysis and downstream task evaluation. Using this benchmark, we systematically evaluate five representative models. Both code and data are publicly released.

Existing audiovisual quality assessment datasets are limited in scale, lack diversity in content and quality degradation types, and provide only holistic scores, thereby hindering research on multimodal perception mechanisms. To address these limitations, this work proposes a crowdsourced subjective evaluation framework that transcends traditional laboratory constraints, integrating a systematic data sampling strategy with a multidimensional annotation scheme. This approach yields YT-NTU-AVQ, the largest and most diverse audiovisual quality assessment dataset to date, comprising 1,620 user-generated videos spanning a broad spectrum of semantic scenarios and quality levels. The dataset and associated platform code have been publicly released, significantly advancing the study and development of multimodal perceptual modeling.

This work addresses the lack of a systematic benchmark for open-domain video shot retrieval, which hinders modeling complex temporal structures and multimodal semantics. To this end, we introduce ShotFinder—the first open-domain shot retrieval benchmark featuring five types of single-factor controllable constraints: temporal dynamics, color, visual style, audio, and resolution. We further propose an “imagination-driven” three-stage retrieval paradigm: (1) video imagination and query expansion using large language models, (2) candidate video recall via search engines, and (3) description-guided temporal localization with multimodal models. Evaluation on 1,210 high-quality YouTube samples reveals that current models significantly underperform humans under color and visual style constraints, highlighting critical gaps in fine-grained semantic alignment and temporal understanding among multimodal foundation models.

This work addresses the high cost and low efficiency associated with constructing high-quality, task-specific video datasets from large-scale video collections. To overcome these challenges, the authors propose an intelligent video retrieval platform that leverages multidimensional semantic modeling to generate deep semantic representations of video clips. The system integrates hybrid retrieval based on natural language queries with a neural re-ranking mechanism to enable precise and efficient construction of customized video subsets. Users can interactively build private retrieval systems through natural language interfaces, and the platform is publicly accessible via a web interface. This approach significantly enhances the efficiency and flexibility of video dataset curation for training, analysis, and evaluation purposes.