Less is More: Improving LLM Alignment via Preference Data Selection

2025-02-20

AI Summary

This work addresses the parameter shrinkage problem in Direct Preference Optimization (DPO) caused by noisy preference data. We propose a dual-margin-guided data selection method that jointly models external reward signals and implicit DPO margins, dynamically selecting highly discriminative samples based on a margin maximization principle. To our knowledge, this is the first work to explicitly formulate data selection as a core optimization dimension in DPO alignment. We further design a lightweight, iterative DPO training framework incorporating this selection mechanism. Experiments demonstrate that using only 10% of the UltraFeedback dataset yields consistent improvements of 3–8% on AlpacaEval 2.0 for Llama and Mistral series models. An iterative training strategy leveraging 25% online data further boosts performance by approximately 3%, while substantially reducing computational overhead compared to standard DPO.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Learning Human Values and Preferences

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Direct Preference Optimization (DPO) has emerged as a promising approach for aligning large language models with human preferences. While prior work mainly extends DPO from the aspect of the objective function, we instead improve DPO from the largely overlooked but critical aspect of data selection. Specifically, we address the issue of parameter shrinkage caused by noisy data by proposing a novel margin-maximization principle for dataset curation in DPO training. To accurately estimate margins for data selection, we propose a dual-margin guided approach that considers both external reward margins and implicit DPO reward margins. Extensive experiments demonstrate that our method reduces computational cost dramatically while improving performance. Remarkably, by using just 10% of the Ultrafeedback dataset, our approach achieves 3% to 8% improvements across various Llama and Mistral series models on the AlpacaEval 2.0 benchmark. Furthermore, our approach seamlessly extends to iterative DPO, yielding a roughly 3% improvement with 25% online data, while further reducing training time. These results highlight the potential of data selection strategies for advancing preference optimization.

Problem

• Improving LLM alignment via data selection
• Reducing parameter shrinkage with margin-maximization
• Enhancing DPO performance with minimal dataset

Innovation

• Margin-maximization principle for DPO
• Dual-margin guided data selection
• Reduces computational cost dramatically