preference learning

This paper addresses the challenge of aligning LLM-based evaluators with human preferences—characterized by miscalibration, sensitivity to scoring criteria, inherent biases, and output instability—by proposing a multi-judge learning framework. Methodologically, it introduces (1) a role-aware preference labeling approach that explicitly models the diversity of human judgments; (2) a unified aggregation framework for multi-judge outputs, incorporating dual-path aggregators—Generalized Additive Models (GAM) and Multi-Layer Perceptrons (MLP)—to enhance robustness; and (3) a score-conditioned judging mechanism that decouples absolute scoring from relative preference estimation. Empirically, the framework achieves significant improvements over baselines in reward modeling and model routing tasks. It demonstrates superior resilience to judge bias and output instability, yielding preference distributions closer to ground-truth human judgments. This work establishes a novel paradigm for trustworthy LLM evaluation.

To address the challenge of efficiently identifying preference-aligned solutions from the Pareto front in multi-objective optimization, this paper proposes a Bayesian preference-based active learning framework. The method models the decision maker’s implicit utility function using pairwise comparison feedback and employs an exploration-exploitation-balanced active sampling strategy, supporting both interactive and posterior usage modes. Its key contributions are: (i) the first systematic integration of Bayesian active learning into high-dimensional (up to nine objectives) multi-objective preference learning, substantially reducing query complexity; and (ii) robust convergence to high-satisfaction solutions with only a small number of pairwise comparisons across multiple benchmark problems. An open-source implementation is provided to facilitate practical adoption and reproducibility.

Existing AI preference alignment models neglect human cognitive mechanisms—such as heuristic simplification—resulting in cognitively implausible and poorly generalizable decision policies. Method: We propose an axiomatic, cognitively faithful decision-making framework: first performing pairwise comparisons over option attributes, then aggregating outcomes via fixed, interpretable rules. Our approach integrates cognitive decision theory with the Bradley–Terry model, constructing a structured information-processing pipeline grounded in empirically observed pairwise comparison data. Contribution/Results: Evaluated on kidney allocation, our model achieves state-of-the-art fidelity to human decisions while substantially improving interpretability and cross-task generalization—demonstrated via rigorous out-of-distribution validation. By grounding value alignment in empirically supported cognitive principles without sacrificing computational tractability, our framework offers a novel, principled pathway toward both cognitive plausibility and engineering feasibility in preference-aligned AI systems.

Existing preference datasets lack explicit modeling of human decision rationales, resulting in inefficient alignment learning, susceptibility to redundant expressions and hallucinations, and prohibitively high annotation costs. This paper proposes a data-centric rationale-augmented paradigm: (1) it is the first to systematically validate the efficacy of free large language models (LLMs) for self-generating decision rationales in preference learning; (2) it introduces a lightweight, general-purpose rationale injection framework that requires no additional human annotation or model fine-tuning and is compatible with mainstream algorithms such as DPO and KTO; and (3) it jointly optimizes rationale-guided contrastive learning and supervised fine-tuning. Experiments demonstrate substantial improvements in data efficiency and training convergence speed, consistent reductions in hallucination rates and redundant outputs across multiple benchmarks, and state-of-the-art performance gains.

Existing post-training methods for generative models—such as RLHF and DPO—rely on pairwise preference comparisons over single samples, limiting their ability to model population-level properties like diversity and bias. This work proposes the first preference optimization framework based on *multi-sample* comparisons, introducing two novel algorithms: mDPO and mIPO. These methods directly optimize collective characteristics of generated outputs at the set level, extending DPO and IPO with intra-group consistency constraints and noise-robust mechanisms. Experiments demonstrate that the proposed framework significantly outperforms single-sample baselines in enhancing output diversity, mitigating bias, and maintaining robustness under label noise. The results validate both the effectiveness and necessity of multi-sample comparison for modeling and optimizing population-level behavioral traits in generative models.

This work addresses the critical challenge of accurately modeling preference functions that aggregate multidimensional criteria into holistic judgments in settings such as admissions and medical diagnosis. Departing from conventional assumptions of linearity or strong structural forms, the paper proposes the first robust nonparametric learning algorithm that achieves optimal performance without requiring any prior knowledge of the preference structure, assuming only monotonic non-decreasing behavior across each criterion. Theoretical analysis demonstrates the severe consequences of common model misspecifications, while experiments on both synthetic and real-world data confirm that the method maintains statistical efficiency under linear preferences and reliably recovers true evaluator preferences in general cases. Notably, the approach effectively uncovers key behavioral differences between human evaluators and large language models in their assessment strategies.

Current large language model alignment relies on ordinal human preferences (e.g., pairwise comparisons), yet theoretical analysis shows such preferences cannot uniquely identify the optimal policy—especially when trade-offs across orthogonal dimensions (e.g., factual accuracy vs. stylistic fluency) are required. Method: This work identifies this as a fundamental limitation and proposes cardinal human feedback, quantifying response quality differences via willingness-to-pay (WTP). We construct a high-fidelity cardinal dataset of 25K samples and integrate it into preference tuning via a novel cardinal loss function. Results: Experiments demonstrate consistent and significant gains over ordinal methods on benchmarks including Arena-Hard, enabling more precise identification of high-impact improvements and enhancing alignment efficiency and robustness. Core contribution: We establish the necessity of cardinal feedback for unambiguous model-level preference identification, thereby advancing beyond the conventional ordinal paradigm in preference learning.

To address the high annotation cost of human feedback in AI alignment and the low statistical efficiency and high redundancy of random pairwise sampling under existing Bradley–Terry models, this paper proposes Swiss InfoGain: a resource-aware preference sample selection method that integrates the Swiss-system tournament mechanism with mutual information gain-driven adaptive pairing. Grounded in game theory, statistical inference, and social choice theory, Swiss InfoGain dynamically prioritizes candidate pairs with maximal information content and highest uncertainty, substantially reducing labeling redundancy. Experiments demonstrate that, under constrained annotation budgets, Swiss InfoGain achieves significantly higher sample efficiency than baseline methods; in high-resource settings, it further improves final model alignment performance and robustness. The core contribution is the first integration of structured tournament mechanisms with information-theoretic criteria into a preference learning sampling framework.

This paper addresses the problem of learning individual heterogeneous preferences from partially observed choice behaviors. Conventional Bradley–Terry–Luce (BTL) models fail to capture preference heterogeneity and nonlinear user-item interactions. To overcome this, we propose a generalized BTL ranking model that represents users and items via low-dimensional latent features and models their interaction through a nonparametric preference function, yielding a score matrix. Methodologically, we design an indirect ℓ∞-regularization framework integrating sieve approximation, low-rank matrix estimation, and one-step Newton debiasing—enabling, for the first time, uncertainty quantification at both aggregate and individual ranking levels. We establish theoretical bounds on estimation error and empirically validate the method on synthetic and real-world datasets, demonstrating high-accuracy score prediction and reliable confidence assessment for rankings. The approach significantly enhances expressive power and statistical interpretability in preference modeling.

This work investigates the fundamental limitations of ordinal preference feedback (e.g., pairwise comparisons) for optimizing large language model outputs in complex human feedback tasks—such as deep research or travel planning. Using the first formal integration of social choice theory (particularly voting theory) into RLHF analysis, and combining it with reinforcement learning theory and preference learning generalization bounds, we rigorously prove that even under ideal conditions—infinite data, zero noise, and online preference acquisition—post-training based solely on ordinal feedback cannot guarantee convergence to an approximately optimal policy. We further disentangle distinct failure modes across reasoning-oriented settings versus instruction tuning, exposing inherent unreliability in ordinal feedback. Our core contribution is establishing a theoretical bottleneck for RLHF in complex reasoning tasks, and demonstrating that overcoming it necessitates incorporating absolute (cardinal) scoring mechanisms and designing novel algorithms grounded in richer feedback structures.