🤖 AI Summary
Current large language model alignment relies on ordinal human preferences (e.g., pairwise comparisons), yet theoretical analysis shows such preferences cannot uniquely identify the optimal policy—especially when trade-offs across orthogonal dimensions (e.g., factual accuracy vs. stylistic fluency) are required. Method: This work identifies this as a fundamental limitation and proposes cardinal human feedback, quantifying response quality differences via willingness-to-pay (WTP). We construct a high-fidelity cardinal dataset of 25K samples and integrate it into preference tuning via a novel cardinal loss function. Results: Experiments demonstrate consistent and significant gains over ordinal methods on benchmarks including Arena-Hard, enabling more precise identification of high-impact improvements and enhancing alignment efficiency and robustness. Core contribution: We establish the necessity of cardinal feedback for unambiguous model-level preference identification, thereby advancing beyond the conventional ordinal paradigm in preference learning.
📝 Abstract
Alignment techniques for LLMs rely on optimizing preference-based objectives -- where these preferences are typically elicited as ordinal, binary choices between responses. Recent work has focused on improving label quality or mitigating particular biases, but we identify a more fundamental limitation: these methods collect the wrong kind of data. We prove an impossibility result: no algorithm relying solely on ordinal comparisons can systematically recover the most preferred model. Intuitively, ordinal data lacks the information needed to resolve tradeoffs -- e.g., fixing a factual error on one prompt versus improving style on another. We show that selecting the optimal model requires recovering preferences over emph{models} (rather than just responses), which can only be identified given cardinal feedback about response quality. To address this, we collect and publicly release a dataset of 25,000 cardinal judgments using willingness-to-pay elicitations, a well-established tool from experimental economics. Empirically, we find that incorporating cardinal feedback into preference fine-tuning allows models to prioritize high-impact improvements and outperform ordinal-only methods on downstream benchmarks, such as Arena-Hard.