DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration

2026-09-25

AI Summary

This study addresses response position bias and systematic misalignment with human preferences in LLM-based evaluation by proposing a unified debiasing and alignment framework. Methodologically, it decouples positional effects from preference structures through latent preference identification. By leveraging large-scale LLM pairwise comparison data alongside limited human annotations, the framework employs an adaptive estimation strategy to balance LLM anchoring signals with human evidence, incorporating fixed-weight uncertainty quantification to enhance robustness. Furthermore, this work constructs a multi-model, dual-order judgment dataset comprising over 410,000 entries. Extensive simulations and benchmark evaluations demonstrate that the proposed approach achieves robust positional debiasing and yields rankings highly consistent with human preferences.

Abstract

Large language models (LLMs) as a judge enable scalable evaluation, but their judgments can be sensitive to response order and, even after removing such position effects, can still diverge systematically from human preferences.We introduce DIAL, a unified framework that combines abundant LLM comparisons with limited human comparisons to separate judge-specific position effects, learn shared structure in position-debiased LLM preferences, and adaptively calibrate that structure toward the human preference target. Theoretically, we study three aspects of DIAL: (i) identification of latent LLM preferences, position effects, and human calibration; (ii) adaptive estimation that balances LLM anchoring against limited human evidence; and (iii) fixed-weight uncertainty quantification for the calibrated human preference. Empirically, we evaluate position debiasing and human alignment separately in controlled simulations and on three human-preference benchmarks, showing that DIAL remains robust to unbalanced response order, achieves strong human-aligned rankings with limited labels, and adapts toward human evidence when LLM information is imperfect. Our real-data study collects over 410K judgments from 21 LLM judges in both display orders, providing a resource for future studies of LLM-judge bias, heterogeneity, and human alignment.

Problem

• LLM-as-a-judge
• position bias
• human preference alignment
• evaluation

Innovation

• Position Debiasing
• Human Preference Calibration
• LLM-as-a-Judge
• Adaptive Estimation
• Uncertainty Quantification