🤖 AI Summary
This work addresses the fragmented landscape of preference learning in large language models, where numerous methods exist without a unifying theoretical foundation, hindering principled practice. We propose the first unified triaxial framework that decomposes existing approaches—such as RLHF, DPO, IPO, KTO, and SimPO—into three orthogonal dimensions: preference modeling, regularization mechanisms, and data distribution. Through theoretical modeling, formal proofs, and extensive empirical validation across more than 50 studies, we delineate the theoretical boundaries between online and offline learning, derive scaling laws governing reward over-optimization, and synthesize actionable guidelines for practitioners. This effort advances preference learning from an empirically driven paradigm toward a theoretically grounded discipline.
📝 Abstract
Aligning large language models (LLMs) with human preferences has become essential for safe and beneficial AI deployment. While Reinforcement Learning from Human Feedback (RLHF) established the dominant paradigm, a proliferation of alternatives -- Direct Preference Optimization (DPO), Identity Preference Optimization (IPO), Kahneman-Tversky Optimization (KTO), Simple Preference Optimization (SimPO), and many others -- has left practitioners without clear guidance on method selection. This survey provides a \textit{theoretical unification} of preference learning methods, revealing that the apparent diversity reduces to principled choices along three orthogonal axes: \textbf{(I) Preference Model} (what likelihood model underlies the objective), \textbf{(II) Regularization Mechanism} (how deviation from reference policies is controlled), and \textbf{(III) Data Distribution} (online vs.\ offline learning and coverage requirements). We formalize each axis with precise definitions and theorems, establishing key results including the coverage separation between online and offline methods, scaling laws for reward overoptimization, and conditions under which direct alignment methods fail. Our analysis reveals that failure modes -- length hacking, mode collapse, likelihood displacement -- arise from specific, predictable combinations of design choices. We synthesize empirical findings across 50+ papers and provide a practitioner's decision guide for method selection. The framework transforms preference learning from an empirical art into a theoretically grounded discipline.