Measuring Human Preferences in RLHF is a Social Science Problem

2026-01-31

AI Summary

Current reinforcement learning from human feedback (RLHF) often misinterprets noisy signals—such as context-dependent judgments, absent genuine attitudes, or ambiguous interpretations—as valid preference data. This work introduces, for the first time, a behavioral science and psychometric perspective by proposing a “preference authenticity spectrum” framework and developing a systematic evaluation methodology based on consistency diagnostics and statistical analysis. Experiments on two widely used RLHF datasets reveal that annotator inconsistency is both systematic and directional; filtering out highly inconsistent annotators reverses the harm classification for 18.6% of prompts and shifts average scores by more than 13 points on a 100-point scale. These findings suggest that existing RLHF approaches may inadvertently model noise rather than authentic human values.

Abstract

RLHF assumes that annotation responses reflect genuine human preferences. We argue this assumption warrants systematic examination, and that behavioral science offers frameworks that bring clarity to when it holds and when it breaks down. Behavioral scientists have documented for sixty years that people routinely produce responses without holding genuine opinions, construct preferences on the spot based on contextual cues, and interpret identical questions differently. These phenomena are pervasive for precisely the value-laden judgments that matter most for alignment, yet this literature has not yet been systematically integrated into ML practice. We argue that the ML community must treat measurement validity as logically prior to preference aggregation. Specifically, we contend that measuring human preferences in RLHF is a social science problem. We present a taxonomy distinguishing genuine preferences from non-attitudes, constructed preferences, and measurement artifacts, along with diagnostic approaches for detecting each. This framework has two important implications. First, it raises the question of whether current RLHF practice may be systematically modeling noise as signal and elicitation artifacts as human values. Second, it provides a path forward by suggesting diagnostic tools that can distinguish valid preferences from artifacts before they enter the training pipeline.

Problem

• RLHF
• human preferences
• measurement validity
• annotation inconsistency
• AI alignment

Innovation

• measurement validity
• preference construction
• annotation inconsistency
• RLHF
• human feedback