Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment

2026-09-20

AI Summary

研究探讨了人类偏好与行为对齐的差异，通过实证分析展示了偏好对齐可能使模型行为更不似人类，揭示了图灵测试差距的存在。

Abstract

Human-feedback alignment has made language models useful assistants and is commonly described as aligning them with humans. However, the responses people prefer from an AI need not be the responses they themselves would give. We distinguish alignment with human preferences from alignment with human behavior, and show that alignment with human preferences can make model behavior less human-like even when both preferences and responses come entirely from humans. We call this the Turing-test gap. We show that preference alignment preserves the human response distribution only under a restrictive condition, and find no consistent evidence that real human preferences satisfy it. Empirically, the loss of human-response likelihood increases with the strength of preference weighting, regardless of its direction, and the gap also appears under standard DPO. These results establish human-likeness as an explicit dimension of alignment rather than something assumed to follow from preference alignment.

Problem

• Human-feedback alignment
• Turing-test gap
• Preference alignment

Innovation

• Turing-test gap
• Human-feedback alignment
• Preference alignment
• Human-likeness