Position Bias Undermines Preference Consistency in Listwise LLM-Based Reranking

2026-08-04

AI Summary

This work addresses the susceptibility of large language models (LLMs) to input order in list reranking, which induces inconsistent preferences and undermines recommendation reliability. The study introduces a multi-level consistency evaluation framework—spanning pairwise preferences, global preference structures, and output stability—to systematically assess the impact of position bias on LLM reranking behavior. Comprehensive experiments across multiple models and datasets reveal that merely enhancing relevance or balancing positional exposure is insufficient to ensure preference consistency. These findings expose an intrinsic instability in LLM-based rerankers that conventional evaluation metrics fail to capture, thereby underscoring the necessity of explicitly modeling preference consistency in reranking systems.

Abstract

Large language models (LLMs) have emerged as promising listwise rerankers for recommender systems, but their reliability under equivalent candidate permutations remains unclear. Since recommendation candidates form an unordered set, a reranker should not depend on the arbitrary order used to serialize them. However, decoder-only LLM rerankers can allow input order to affect model scores, pairwise preferences, and rankings. We study how position bias affects the ranking process induced by LLM-based rerankers. Instead of measuring only changes in final ranked lists, we treat rankings produced under equivalent candidate permutations as observations of an induced preference system. We introduce an evaluation framework measuring pairwise preference instability, global preference inconsistency, and listwise output consistency. This framework characterizes candidate-order sensitivity at the pairwise, global, and output levels. Experiments across multiple LLMs, datasets, and list lengths show that these consistency measures are closely aligned, but can diverge from recommendation effectiveness and marginal position-exposure bias. Improving relevance or flattening exposure across positions does not necessarily restore stable pairwise preferences, globally coherent preference structures, or consistent ranked outputs. These results show that reducing marginal exposure skew is insufficient to establish ranking-function validity in LLM-based reranking. Code is available at https://github.com/ejbito/InvariRank .

Problem

• position bias
• preference consistency
• LLM-based reranking
• listwise ranking
• candidate permutation

Innovation

• position bias
• preference consistency
• listwise reranking
• LLM-based recommender
• order invariance