Hummer: Towards Limited Competitive Preference Dataset

2024-05-19

arXiv.org

AI Summary

Existing preference datasets suffer from intrinsic conflicts among multiple alignment objectives, rendering models vulnerable to jailbreaking attacks and hindering prioritized adaptation to downstream tasks. To address this, we propose the Alignment Dimension Conflict (ADC) metric—a novel statistical measure that quantifies inter-objective alignment conflict for the first time. We further introduce Hummer, the first low-conflict preference dataset explicitly designed for alignment decoupling; it comprises two variants—Hummer (hybrid-sampled from UltraFeedback and GPT-4 AI feedback) and its fine-grained subset Hummer-F. Accompanying this dataset, we train dedicated reward models: HummerRM and HummerRM-F. Experiments demonstrate that Hummer significantly reduces intra-dataset alignment conflict, markedly improving reward model robustness, cross-domain transferability, and resistance to jailbreaking. Our work establishes a new paradigm for multi-objective alignment modeling grounded in conflict-aware dataset curation and decoupled objective learning.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Preference datasets are essential for incorporating human preferences into pre-trained language models, playing a key role in the success of Reinforcement Learning from Human Feedback. However, these datasets often demonstrate conflicting alignment objectives, leading to increased vulnerability to jailbreak attacks and challenges in adapting downstream tasks to prioritize specific alignment objectives without negatively impacting others. In this work, we introduce a novel statistical metric, Alignment Dimension Conflict, to quantify the degree of conflict within preference datasets. We then present exttt{Hummer} and its fine-grained variant, exttt{Hummer-F}, as innovative pairwise preference datasets with reduced-conflict alignment objectives. exttt{Hummer} is built based on UltraFeedback and is enhanced by AI feedback from GPT-4, marking as the first preference dataset aimed at reducing the competition between alignment objectives. Furthermore, we develop reward models, HummerRM and HummerRM-F, which employ a hybrid sampling approach to balance diverse alignment objectives effectively. This sampling method positions HummerRM as an ideal model for domain-specific further fine-tuning and reducing vulnerabilities to attacks.

Problem

• Quantifying conflicts in preference datasets' alignment objectives
• Developing reduced-conflict pairwise preference datasets
• Creating reward models balancing diverse alignment objectives

Innovation

• Introduced Alignment Dimension Conflict metric
• Created Hummer datasets with reduced-conflict objectives
• Developed hybrid sampling reward models