HPS: Hard Preference Sampling for Human Preference Alignment

2025-02-20

AI Summary

To address challenges in aligning large language models (LLMs) with human preferences—including difficulty rejecting harmful content, inefficient utilization of negative samples, and high computational overhead—this paper proposes the Hard Preference Sampling (HPS) framework. HPS introduces a novel “hard-negative-first rejection” mechanism, leveraging Plackett–Luce theory to guide single-sample Monte Carlo sampling for identifying the most ambiguous (i.e., hardest-to-distinguish) negative responses. It further designs a hard-negative-weighted loss and a joint preference-antipreference optimization objective. Theoretically, HPS improves sample efficiency and enlarges the reward margin. Empirical evaluation on HH-RLHF and PKU-Safety demonstrates that HPS significantly increases reward margins, substantially reduces harmful outputs, and achieves state-of-the-art performance in both BLEU score and reward modeling accuracy—thereby simultaneously enhancing safety, alignment quality, and training efficiency.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Natural Language Processing: Safety and Robustness
• Machine Learning: Learning Preferences or Rankings

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Aligning Large Language Model (LLM) responses with human preferences is vital for building safe and controllable AI systems. While preference optimization methods based on Plackett-Luce (PL) and Bradley-Terry (BT) models have shown promise, they face challenges such as poor handling of harmful content, inefficient use of dispreferred responses, and, specifically for PL, high computational costs. To address these issues, we propose Hard Preference Sampling (HPS), a novel framework for robust and efficient human preference alignment. HPS introduces a training loss that prioritizes the most preferred response while rejecting all dispreferred and harmful ones. It emphasizes"hard"dispreferred responses--those closely resembling preferred ones--to enhance the model's rejection capabilities. By leveraging a single-sample Monte Carlo sampling strategy, HPS reduces computational overhead while maintaining alignment quality. Theoretically, HPS improves sample efficiency over existing PL methods and maximizes the reward margin between preferred and dispreferred responses, ensuring clearer distinctions. Experiments on HH-RLHF and PKU-Safety datasets validate HPS's effectiveness, achieving comparable BLEU and reward scores while greatly improving reward margins and thus reducing harmful content generation.

Problem

• Align LLM responses with human preferences
• Improve handling of harmful content
• Reduce computational costs efficiently

Innovation

• Hard Preference Sampling framework
• Single-sample Monte Carlo strategy
• Emphasis on hard dispreferred responses