DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes

2026-07-27

AI Summary

This work addresses the heuristic-driven, non-systematic, and irreproducible nature of data mixing in vision-language model pretraining by decoupling data recipe construction into two subproblems: inter-capability ratio search and intra-category data allocation. The former employs univariate iterative search to identify optimal capability ratios, while the latter selects data through constrained convex optimization incorporating diversity objectives, guided by quality and difficulty scores. This approach establishes, for the first time, a reproducible and interpretable engineering pipeline for data mixing and enables seamless scaling from small-scale optimal recipes to large-scale training. Using only 80 billion multimodal tokens for pretraining, the resulting model matches the performance of state-of-the-art open-source models trained with substantially larger data budgets and significantly outperforms heuristic baselines.

Abstract

While data curation for Vision Language Models (VLMs) is increasingly active, public practice for constructing pretraining mixtures remains largely heuristic: practitioners stack datasets that pass quality filters, set cross-domain ratios by intuition, and lack a principled, attributable criterion for admitting new data, while frontier recipes remain undisclosed. We formulate data construction as a systematic mixture-optimization problem and turn it into a reproducible engineering discipline by decoupling the mixture into two orthogonal sub-problems: inter-class ratios across capabilities and intra-class ratios within a category. For inter-class allocation, we use a single-variable iterative search; for intra-class composition, we apply a multidimensional, dataset-level assessment scoring Quality and Difficulty, and formulate selection as a constrained convex optimization with a diversity objective. The DecoupleMix framework delivers two critical capabilities: guiding what data to collect next and rendering dataset validation a controlled, attributable experiment. Experiments show our approach consistently surpasses heuristic baselines. Moreover, optimal ratios discovered on small-scale proxies transfer seamlessly to larger scales without retuning. Using 80B additional multimodal continue-pretraining tokens, our VLM is competitive with strong open-source models trained with substantially larger multimodal budgets.

Problem

• Vision Language Models
• data curation
• pretraining mixtures
• heuristic methods
• data recipes

Innovation

• DecoupleMix
• mixture optimization
• convex allocation
• data curation
• vision-language models