Private Direct Preference Optimization for LLM Alignment

2026-08-05

AI Summary

This work addresses the privacy risks in large language model alignment, where relative human preference data may leak sensitive attributes. Existing differentially private methods introduce excessive noise and bias due to their incompatibility with the structure of preference signals. To overcome this, the paper proposes PrivDPO, which formalizes the novel concept of “preference privacy” and integrates it into the Direct Preference Optimization (DPO) framework. PrivDPO protects only the one-dimensional preference signal by identifying that gradient differences flow along a text-determined preference axis, and introduces an unbiased stochastic rescaling mechanism that avoids per-sample perturbation. Evaluated across three alignment benchmarks and three large language models, PrivDPO significantly outperforms current privacy-preserving baselines, achieving a superior utility–privacy trade-off under strict privacy guarantees.

Abstract

Direct preference optimization (DPO) is now a standard method for aligning large language models (LLMs) using human preference data. Each DPO example contains a prompt and a pair of candidate model responses. While prompts and responses are often public or model-generated, the relative preference between responses reflects subjective judgments and can reveal sensitive attributes of annotators or end users. Off-the-shelf privacy-preserving approaches are not well matched to this structure, leading to unnecessary noise injection and biased updates in training. In this paper, we formalize preference privacy, a label-DP-style privacy notion for DPO that protects only the relative preference between candidate responses, assuming an adversary who already knows the prompt and responses. We then design PrivDPO, a DPO variant that enforces preference privacy while remaining compatible with large-scale LLM training. Our main observation is that, for neighboring examples differing only in their preference signal, the gradient difference lies on a one-dimensional preference axis determined solely by the text; all preference information flows through this axis. PrivDPO adds calibrated randomness only along this axis via an unbiased randomized rescaling of the DPO objective, avoiding per-example gradient operations. Our experiments on three alignment benchmarks and three LLM families show that PrivDPO consistently achieves strong privacy-utility trade-offs compared with privacy-preserving baselines.

Problem

• preference privacy
• direct preference optimization
• differential privacy
• LLM alignment
• sensitive attributes

Innovation

• preference privacy
• Direct Preference Optimization
• differential privacy
• gradient rescaling
• LLM alignment