Ranking with Confidence for Large Scale Comparison Data

2022-02-03

arXiv.org

AI Summary

To address the challenge of learning total orders from large-scale, sparse, and highly noisy pairwise comparison data, this paper proposes a generative, noise-aware ranking algorithm. Methodologically, it explicitly incorporates pairwise comparison confidence into a scalable ranking framework for the first time; introduces a quasiconvex approximation strategy for the underlying nonconvex optimization problem, solved via iterative reweighted minimization combined with the Primal-Dual Hybrid Gradient method; and extends the Bradley–Terry model to capture heterogeneous noise. Empirically, the method achieves a 0.1 improvement in Kendall tau over state-of-the-art approaches, maintains robustness under 10% erroneous comparisons, and reduces computational time by an order of magnitude—enabling ranking in seconds. Validation on real-world datasets demonstrates significant superiority over active learning baselines. Overall, the approach establishes a new Pareto-optimal trade-off between accuracy and efficiency in large-scale noisy ranking.

Abstract

In this work, we leverage a generative data model considering comparison noise to develop a fast, precise, and informative ranking algorithm from pairwise comparisons that produces a measure of confidence on each comparison. The problem of ranking a large number of items from noisy and sparse pairwise comparison data arises in diverse applications, like ranking players in online games, document retrieval or ranking human perceptions. Although different algorithms are available, we need fast, large-scale algorithms whose accuracy degrades gracefully when the number of comparisons is too small. Fitting our proposed model entails solving a non-convex optimization problem, which we tightly approximate by a sum of quasi-convex functions and a regularization term. Resorting to an iterative reweighted minimization and the Primal-Dual Hybrid Gradient method, we obtain PD-Rank, achieving a Kendall tau 0.1 higher than all comparing methods, even for 10% of wrong comparisons in simulated data matching our data model, and leading in accuracy if data is generated according to the Bradley-Terry model, in both cases faster by one order of magnitude, in seconds. In real data, PD-Rank requires less computational time to achieve the same Kendall tau than active learning methods.

Problem

• Ranking Algorithm
• Large-scale Data
• Confidence Estimation

Innovation

• PD-Rank
• Efficiency
• Accuracy