sequential dpo

Direct Preference Optimization (DPO) lacks a systematic taxonomy, standardized evaluation protocols, and unified theoretical foundations. Method: We propose the first four-dimensional DPO taxonomy—spanning data strategies, learning frameworks, constraint mechanisms, and model attributes—and establish a standardized empirical evaluation framework, conducting cross-method comparative analyses across multiple benchmarks. We further release an open-source, continuously updated DPO repository encompassing code, datasets, and reproducible scripts. Contribution/Results: This work achieves the first unified taxonomy, standardized evaluation, and fully reproducible implementation in DPO research. It provides both a theoretical framework and practical engineering guidelines for DPO, significantly enhancing the robustness and generalization of LLM alignment methods. By enabling rigorous, comparable, and reproducible experimentation, our contributions advance trustworthy and efficient human preference modeling.

DPO, proposed as a lightweight alternative to RLHF, empirically underperforms PPO-RLHF in preference alignment. This work identifies—through systematic analysis—a fundamental flaw in DPO: severe imbalance in gradient contributions from preference pairs, which destabilizes optimization trajectories and leads to suboptimal convergence. To address this, we propose Balanced-DPO, a theoretically grounded, implementation-light gradient reweighting method that requires no auxiliary models, additional data, or hyperparameter tuning. Derived from sensitivity analysis of the DPO objective’s gradients, Balanced-DPO seamlessly integrates into standard supervised fine-tuning pipelines. Evaluated across multiple LLM preference alignment benchmarks, Balanced-DPO consistently outperforms vanilla DPO (average win rate +3.2%), exhibits improved training stability, and substantially narrows the performance gap with PPO-RLHF.

This work addresses a key limitation of existing Direct Preference Optimization (DPO) methods, which rely solely on pairwise preference signals and neglect the quantitative differences in response quality, leading to ambiguous training signals and suboptimal optimization efficiency. To overcome this, we propose a novel preference optimization algorithm that, for the first time, incorporates explicit score gaps into the DPO framework. Our approach preserves the advantage of not requiring an explicit reward model while leveraging fine-grained relative quality information to enhance alignment. By designing a loss function that accounts for score differences, the method enjoys faster theoretical statistical convergence and demonstrates robustness to scoring noise. Extensive experiments show consistent and significant improvements over current DPO variants across multiple large language models and evaluation benchmarks, with stable performance gains even when provided with inaccurate scores.

Existing DPO variants lack rigorous attribution analysis and fair comparative evaluation of their improvement components, hindering identification of genuinely effective technical pathways. Method: We propose the first unified preference optimization framework that systematically decomposes mainstream DPO enhancements into seven orthogonal dimensions—temperature scaling, reward normalization, dynamic margin, symmetric loss, top-k sampling, gradient reweighting, and multi-turn feedback modeling—and integrates them via a unified objective function to enable synergistic interaction. Contribution/Results: Our framework enables modular composition and quantitative attribution analysis for the first time. It substantially outperforms DPO, IPOL, KTO, and other baselines across multiple benchmarks, validating the efficacy of integrated strategies. Furthermore, we open-source a reusable implementation and practical guidelines to advance standardization and reproducibility in preference optimization research.

DPO in language model post-training suffers from implicit reward overfitting and divergence, leading to policy degradation—where even preferred responses approach zero probability. This paper identifies the root cause as implicit reward over-adaptation to preference data. To address this, we propose Reward-Distilled DPO (RD-DPO), the first DPO variant integrating explicit reward model distillation: it jointly optimizes a family of reward models to calibrate the language model’s implicit reward distribution. RD-DPO unifies implicit reward modeling, reward knowledge distillation, and multi-model ensembling, preserving DPO’s inference efficiency and simplicity without added computational overhead. Experiments demonstrate that RD-DPO significantly mitigates policy degradation and enhances alignment stability and generalization robustness under distributional shift.

This work addresses two key limitations of Direct Preference Optimization (DPO) in multi-turn dialogue agent tasks: (1) bias arising from the intractable partition function, and (2) modeling inaccuracies due to inconsistent trajectory lengths between preferred and dispreferred responses. To this end, we propose Distribution-Matching Preference Optimization (DMPO). Methodologically, DMPO reformulates the RL objective via distribution matching over state-action occupancy measures—replacing conventional policy constraints to ensure theoretical soundness—and incorporates trajectory-length normalization into the Bradley–Terry preference model to mitigate length-induced bias. Empirically, DMPO achieves significant improvements over existing DPO variants across three multi-turn dialogue agent benchmarks, demonstrating enhanced training stability, cross-task generalization, and optimization efficiency.

This work addresses a limitation in existing offline preference optimization methods, such as Direct Preference Optimization (DPO), which utilize only the chosen and rejected responses from static datasets while neglecting the greedy response generated by the reference model for the same prompt as a potential supervisory signal. The authors propose DPOP, which extends the DPO framework by introducing a gated penalty term that suppresses the reference model’s greedy response only when the policy model assigns lower likelihood to the preferred response than to the rejected one. Combined with length normalization to enhance fairness, this approach uniquely leverages the reference model’s own greedy output as a conditionally activated supervision signal, substantially improving preference learning. On AlpacaEval 2.0, DPOP achieves length-controlled win rate improvements of 5.3% and 4.4% over baselines using Llama-3-8b-instruct and Gemma-2-9b-instruct, respectively.

This work establishes that the equivalence between Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF) hinges on a commonly violated implicit assumption: that the optimal RLHF policy must strictly prefer human-preferred responses. When this assumption fails, DPO merely optimizes relative advantages over a reference policy, potentially leading to pathological convergence rather than genuine alignment with human preferences. To address this, we propose Constrained Preference Optimization (CPO), a framework that retains simplicity while offering provable alignment guarantees. Through theoretical analysis, geometric interpretation via soft-margin ranking, constrained optimization, and large-scale experiments, we demonstrate that CPO achieves state-of-the-art performance on standard benchmarks. Our work also formally characterizes the conditions under which DPO and RLHF are equivalent, clarifying both the validity regime and failure modes of DPO.

This work addresses the gradient asymmetry inherent in Direct Preference Optimization (DPO), which biases models toward avoiding incorrect responses rather than proactively generating high-quality ones. To remedy this, the authors propose AdaDPO, an adaptive variant of DPO that explicitly corrects gradient imbalance at the loss level without altering data or model architecture. AdaDPO dynamically computes per-sample coefficients based on the policy model’s generation probabilities and integrates stop-gradient operations, token-level gradient balancing, numerical clipping, and adaptive weighting. It is compatible with various contrastive preference losses, including SimPO, IPO, and CPO. Evaluated on Llama-3-8B-Instruct, AdaDPO substantially outperforms standard DPO, achieving a 48.3% win rate on AlpacaEval 2 under length control and surpassing DPO in 81% of hyperparameter configurations, while effectively mitigating length bias.

This study investigates whether sequential preference optimization leads to uniform forgetting of previously learned preferences and how this phenomenon is influenced by the relationships among preference objectives. Using Llama-3.1-8B-Instruct with LoRA adapters, the authors apply Direct Preference Optimization (DPO) sequentially across four distinct preference settings, employing length-normalized margins and quartile-based decomposition for fine-grained analysis alongside gradient diagnostics. The findings reveal that sequential DPO does not induce uniform forgetting; instead, it exhibits diverse behaviors ranging from degradation and stability to positive transfer. Objective compatibility and signal strength emerge as key determinants of these dynamics. High-confidence preference pairs can either improve or deteriorate across stages, and inter-stage gradients are nearly orthogonal, suggesting that gradient interference is not the primary cause of forgetting. These insights offer new design principles for multi-objective alignment.