Detecting Prefix Bias in LLM-based Reward Models

2025-05-13

AI Summary

This work identifies and systematically evaluates “prefix bias” in large language model (LLM) reward models—i.e., spurious shifts in racial/gender preferences induced by minor syntactic variations in query prefixes (e.g., honorifics, phrasing). We propose the first scalable detection framework, leveraging controlled prefix perturbations, sensitivity analysis, and multidimensional fairness quantification; it demonstrates broad applicability and statistical significance across multiple open-source preference datasets and state-of-the-art reward models. To mitigate this bias, we design an adversarial, prefix-aware data augmentation strategy. Experiments show it reduces bias magnitude by 37.2% on average, substantially improving fairness and robustness. Our core contributions are: (1) formal definition of prefix bias as a novel, linguistically grounded fairness vulnerability; (2) establishment of the first standardized benchmark for its detection; and (3) provision of a practical, deployable mitigation method validated across diverse reward modeling settings.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Reinforcement Learning with Human Feedback (RLHF) has emerged as a key paradigm for task-specific fine-tuning of language models using human preference data. While numerous publicly available preference datasets provide pairwise comparisons of responses, the potential for biases in the resulting reward models remains underexplored. In this work, we introduce novel methods to detect and evaluate prefix bias -- a systematic shift in model preferences triggered by minor variations in query prefixes -- in LLM-based reward models trained on such datasets. We leverage these metrics to reveal significant biases in preference models across racial and gender dimensions. Our comprehensive evaluation spans diverse open-source preference datasets and reward model architectures, demonstrating susceptibility to this kind of bias regardless of the underlying model architecture. Furthermore, we propose a data augmentation strategy to mitigate these biases, showing its effectiveness in reducing the impact of prefix bias. Our findings highlight the critical need for bias-aware dataset design and evaluation in developing fair and reliable reward models, contributing to the broader discourse on fairness in AI.

Problem

• Detecting prefix bias in LLM-based reward models
• Evaluating racial and gender biases in preference models
• Mitigating biases through data augmentation strategies

Innovation

• Detect prefix bias in LLM reward models
• Evaluate bias across racial and gender dimensions
• Propose data augmentation to mitigate prefix bias