PAFO: Pareto Fairness Optimization for Personalized Reward Modeling

2026-06-06

AI Summary

This work addresses the systematic bias of personalized reward models against minority user groups under imbalanced preference data, which leads to skewed reward quality. To mitigate this issue, the study introduces Pareto fairness optimization into this setting and proposes a unified model architecture that leverages group labels only during training. The approach first trains separate reward models for majority and minority groups, then distills their preference boundaries into a single model via conditional boundary supervision, eliminating the need for group information at inference time. Evaluated on the Personal-LLM and DSP datasets, the method simultaneously improves prediction accuracy for both majority and minority users while significantly reducing multiple user-level unfairness metrics.

Abstract

Large language models (LLMs) increasingly rely on reward models to align their outputs with diverse user preferences. While personalized reward models aim to capture such heterogeneity, they are often trained on imbalanced user preference data and may therefore favor users whose preferences are more common in the training population. In this paper, we identify this failure mode as personalized reward bias, where reward modeling quality varies systematically with preference support rate. We formulate its mitigation as a Pareto fairness problem over group utilities, aiming to improve under-served users without degrading other user groups. To this end, we propose PAFO, a Pareto fairness optimization framework for personalized reward modeling. PAFO first trains group-specialized reward models for majority and minority preference groups, then constructs conditional margin-level supervision to distill their heterogeneous preference boundaries into a single unified model. The resulting model uses group information only during training and requires no explicit group labels at inference time. Experiments on Personal-LLM and DSP show that PAFO improves both minority-group and majority-group accuracy while reducing user-level unfairness across multiple metrics, demonstrating its effectiveness for fairer LLM personalization.

Problem

• personalized reward bias
• preference imbalance
• Pareto fairness
• user-level unfairness
• reward modeling

Innovation

• Pareto fairness
• personalized reward modeling
• reward bias mitigation
• preference distillation
• fair LLM personalization