Learning when to rank: Estimation of partial rankings from sparse, noisy comparisons

2025-01-05

AI Summary

To address the problem of unreasonable total rankings induced by forcing a strict linear order on sparse and noisy pairwise comparison data, this paper proposes a Bayesian partial ranking method that establishes, for the first time, a provably evidence-driven framework: it adaptively infers ties based on empirical support strength, thereby avoiding spurious strict ordering when evidence is insufficient. The method integrates Bayesian inference with maximum a posteriori (MAP) estimation, jointly coupling adaptive agglomerative clustering with a Bradley–Terry–type generative model. This yields a modular, plug-and-play paradigm compatible with arbitrary score- or ranking-based models. Extensive evaluation on multiple real-world and synthetic network datasets demonstrates that the approach significantly enhances both interpretability and robustness of rankings—particularly under low observation density—by producing more concise and statistically credible partial-order summaries.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Reasoning under Uncertainty: Relational Probabilistic Models
• Knowledge Representation and Reasoning: Preferences

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

A common task arising in various domains is that of ranking items based on the outcomes of pairwise comparisons, from ranking players and teams in sports to ranking products or brands in marketing studies and recommendation systems. Statistical inference-based methods such as the Bradley-Terry model, which extract rankings based on an underlying generative model of the comparison outcomes, have emerged as flexible and powerful tools to tackle the task of ranking in empirical data. In situations with limited and/or noisy comparisons, it is often challenging to confidently distinguish the performance of different items based on the evidence available in the data. However, existing inference-based ranking methods overwhelmingly choose to assign each item to a unique rank or score, suggesting a meaningful distinction when there is none. Here, we address this problem by developing a principled Bayesian methodology for learning partial rankings -- rankings with ties -- that distinguishes among the ranks of different items only when there is sufficient evidence available in the data. Our framework is adaptable to any statistical ranking method in which the outcomes of pairwise observations depend on the ranks or scores of the items being compared. We develop a fast agglomerative algorithm to perform Maximum A Posteriori (MAP) inference of partial rankings under our framework and examine the performance of our method on a variety of real and synthetic network datasets, finding that it frequently gives a more parsimonious summary of the data than traditional ranking, particularly when observations are sparse.

Problem

• Ranking
• Incomplete Information
• Pairwise Comparison

Innovation

• Bayesian Approach
• Partial Ranking
• Efficient Algorithm