🤖 AI Summary
This work addresses the challenge of efficiently selecting high-quality subsets from ultra-large corpora to reduce fine-tuning costs. The authors propose CRAFT, a method that decomposes the source–target joint distribution and employs a two-stage strategy: first allocating the selection budget across clusters according to k-means proportions to match the source distribution of the validation set, then selecting within each cluster samples whose target embeddings best approximate the validation target distribution. CRAFT is the first approach to combine clustering with conditional expectation distance for data selection. Theoretically, proportional allocation bounds the KL divergence regardless of the specific embedding scheme. Evaluated on fine-tuning mBART with 33 million sentence pairs, CRAFT achieves a BLEU score of 43.34—outperforming TSDS by 2.13 points—and runs 2.8× faster than TAROT, completing the entire pipeline on CPU in under one minute.
📝 Abstract
Selecting a small, high-quality subset from a large corpus for fine-tuning is increasingly important as corpora grow to tens of millions of datapoints, making full fine-tuning expensive and often unnecessary. We propose CRAFT (Clustered Regression for Adaptive Filtering of Training data), a vectorization-agnostic selection method for training sequence-to-sequence models. CRAFT decomposes the joint source-target distribution and performs a two-stage selection: (i) match the validation source distribution through proportional budget allocation across k-means clusters, and (ii) within each source cluster, select training pairs whose target embeddings minimize a conditional expected distance derived from the validation target distribution. We prove that proportional cluster allocation bounds the continuous KL divergence between selected and validation distributions, with the residual controlled by cluster diameters. We evaluate CRAFT on English-Hindi translation by selecting training data from 33 million NLLB sentence pairs and fine-tuning mBART via LoRA. CRAFT achieves 43.34 BLEU, outperforming TSDS (41.21) by 2.13 points on the same candidate pool and encoder while completing selection over 40 times faster. With TF-IDF vectorization, the entire pipeline completes in under one minute on CPU. TAROT achieves 45.61 BLEU, but CRAFT completes selection in 26.86 seconds versus TAROT's 75.6 seconds, a 2.8 time speedup.