🤖 AI Summary
This work addresses the limitations of existing meta-learning training data selection (MTS) methods, which suffer from performance degradation due to the mismatch between synthetic and real data distributions. For the first time, it identifies the underlying failure mechanisms of MTS through the lenses of low gradient signal-to-noise ratio and insufficient feature informativeness. To overcome these issues without requiring complex architectural modifications, the study proposes an efficient strategy that enhances gradient quality by increasing batch size and introduces a novel data quality metric to improve selection efficacy. Built upon a bilevel optimization framework, the method jointly models data distributional positioning and training dynamics. Experiments across four benchmarks demonstrate average performance gains of 5.49% over non-selection baselines and 2.89% over the strongest existing MTS approach, substantially improving the utilization efficiency of synthetic data.
📝 Abstract
Synthetic data are increasingly used to train neural networks, yet distributional mismatch with real data limits their effectiveness when used indiscriminately. A common strategy is to learn data weights via bi-level optimization, which we refer to as Meta-learning for Training-data Selection (MTS). Interestingly, in practice, MTS often performs below expectation. We identify two obstacles in properly training MTS: a poor gradient signal-to-noise ratio (GSNR), which causes optimization difficulties, and lack of informative features that correlates with data quality. We present a mathematical analysis of MTS, which reveals the dynamics of normalized data weights and the relation between disparate data quality and poor GSNR. The analysis suggests a a simple yet effective solution: increasing the batch size. Further, we propose a set of informative features that capture the positions of training data in their distributions and training dynamics. Experiments across four benchmarks show consistent improvements, achieving average gains of 5.49% over training without selection and 2.89% over the strongest baseline.