What Do People Actually Want From AI? Mapping Preference Plurality

2026-06-04

AI Summary

Current AI alignment approaches, such as reinforcement learning from human feedback (RLHF), reduce human preferences to a unified reward signal, thereby overlooking their inherent diversity, context dependence, and semantic disagreements, which limits their capacity to authentically capture user needs. This study leverages 1,500 open-ended responses from participants across 75 countries, conducting the first large-scale cross-cultural qualitative analysis by applying thematic coding and semantic parsing to the PRISM dataset—moving beyond conventional pairwise comparison paradigms. Findings reveal that only 49% of users referenced “truthfulness,” with highly heterogeneous definitions; most values garnered support from fewer than 25% of respondents; and features like anthropomorphism and safety guardrails elicited significant controversy. These results expose fundamental shortcomings in existing alignment mechanisms in addressing value pluralism and semantic conflict, critiquing monolithic reward modeling as a form of epistemic violence.

Abstract

Large Language Models (LLMs) are often fine-tuned through Reinforcement Learning from Human Feedback (RLHF) to align with people's preferences and values. However, this method has known limitations: it aggregates conflicting preferences, often relies on unrepresentative samples, and uses only binary comparisons. Analysing 1,500 open-ended responses from the PRISM dataset across 75 countries, we examine what people actually want from AI systems and reveal concrete failures of current methods. We find that different people want different things: most values are requested by fewer than a quarter of respondents, with truthfulness the sole exception at 49%. Furthermore, the same words hide divergent meanings: when people describe what they mean by "truthfulness", they reveal distinct, potentially incompatible, epistemological bases, as some ask for sourced claims, some for expert opinions, and some even ask for unpopular views. Certain capabilities, namely how human-like a model behaves, and some features, like AI guardrails, are outright controversial, with some desiring them and others rejecting them. We additionally find that people often use contextual distinctions (what AI should do "by default" versus "if requested") that binary comparisons cannot capture. These findings expose fundamental problems in current alignment practices. When 49% request truthfulness but define it differently, this is unlikely to be captured by a single reward model. The persistence of high hallucination rates in well-funded models, despite users' clear demands for accuracy, suggests that current methods fail to identify actual preferences. This paper sheds light on the situated, contested, imperfect signals that are currently being flattened into universal preference models, a practice others have characterised as epistemic violence.

Problem

• preference plurality
• AI alignment
• human feedback
• value disagreement
• epistemic diversity

Innovation

• preference plurality
• value alignment
• epistemic diversity
• RLHF limitations
• contextual distinctions