PIPA: Preference Alignment as Prior-Informed Statistical Estimation

2025-02-09

AI Summary

Offline preference alignment lacks a unified modeling framework capable of jointly handling paired and unpaired data, as well as answer-level and step-level annotations. This paper proposes PIPA—the first reinforcement-learning-free (RL-free) unified probabilistic framework for preference alignment, formulated as maximum likelihood estimation with customizable prior constraints. Theoretically, we show that both DPO and KTO emerge as special cases of PIPA under specific prior choices. By introducing a modular Bayesian prior mechanism, we derive two novel variants: PIPA-M (incorporating a matching prior) and PIPA-N (employing a normalization prior). Evaluated on GSM8K and MATH benchmarks, PIPA achieves 3–10% absolute performance gains over strong baselines, without incurring additional training overhead or computational cost. Our framework significantly enhances flexibility and generalizability in preference modeling across diverse data regimes and annotation granularities.

Technology Category

• Knowledge Representation and Reasoning: Preferences
• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Learning Human Values and Preferences

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Offline preference alignment for language models such as Direct Preference Optimization (DPO) is favored for its effectiveness and simplicity, eliminating the need for costly reinforcement learning. Various offline algorithms have been developed for different data settings, yet they lack a unified understanding. In this study, we introduce Pior-Informed Preference Alignment (PIPA), a unified, RL-free probabilistic framework that formulates language model preference alignment as a Maximum Likelihood Estimation (MLE) problem with prior constraints. This method effectively accommodates both paired and unpaired data, as well as answer and step-level annotations. We illustrate that DPO and KTO are special cases with different prior constraints within our framework. By integrating different types of prior information, we developed two variations of PIPA: PIPA-M and PIPA-N. Both algorithms demonstrate a $3sim10%$ performance enhancement on the GSM8K and MATH benchmarks across all configurations, achieving these gains without additional training or computational costs compared to existing algorithms.

Problem

• Unifies offline preference alignment algorithms
• Accommodates paired and unpaired data types
• Enhances performance without extra computational cost

Innovation

• RL-free probabilistic framework
• Maximum Likelihood Estimation
• Prior-informed preference alignment