🤖 AI Summary
This work proposes a systematic data engineering methodology to address the engineering challenges of processing and managing raw video data for large-scale video foundation model training. By leveraging metadata-driven data cleaning, multi-stage filtering, and an inference-aware architecture—combined with the Lavender Data management system, μP hyperparameter transfer, and hyperspherical geometric constraint optimization—the authors efficiently construct a high-quality training set from approximately 50 million video clips. Using this curated dataset, they successfully train Summer-22B, a 22-billion-parameter video foundation model. This study provides the first empirical validation that systematic data engineering plays a dominant role in enhancing model performance, underscoring the critical importance of data quality and structural integrity in large-scale video modeling.
📝 Abstract
We describe our experience training Summer-22B, a video foundation model developed from scratch. This report documents the engineering challenges, design decisions, and lessons learned while scaling from raw footage collection to a functional model trained on approximately 50 million clips. We outline our approach combining metadata-driven dataset curation, multi-stage filtering, $μ$P parameterization, and hypersphere-constrained optimization. We developed the Lavender Data system for dataset management and adopted inference-aware architectural choices. We share observations on what worked in our setting: dataset engineering consumed the majority of effort, architectural variants showed smaller differences than we expected, and $μ$P hyperparameter transfer appeared effective even under geometric constraints. We hope this account proves useful to others undertaking similar projects.