Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation

2026-06-07

AI Summary

This work addresses a critical limitation in current model leaderboard evaluation methods, which often overlook task-level performance uncertainty and variability, thereby yielding unreliable global rankings. To remedy this, the authors propose a hierarchical framework that enables principled aggregation of uncertainty from the task level to the leaderboard level. By integrating task-wise pairwise comparisons with conformal prediction, the method produces statistically valid confidence intervals for both individual task rankings and overall model rankings. Empirical validation on synthetic data as well as real-world benchmarks—TabArena and PromptEval (MMLU)—demonstrates the approach’s statistical validity and informativeness. The resulting rankings are not only reliable but also explicitly account for uncertainty, offering a robust foundation for evaluating and comparing models on new tasks.

Abstract

Pretrained models are often evaluated on multi-task leaderboards to measure their applicability in diverse contexts. However, current methods for aggregating performance across tasks into leaderboard-level rankings do not address the uncertainty and variability at the task level. While recent works have proposed interval-based model rankings, the principled aggregation of uncertainty from individual tasks to leaderboard-level rankings remains unaddressed, and variation in models' performance across tasks is frequently obscured. In this work, we introduce a hierarchical framework that constructs model rank intervals with statistical guarantees at both levels: task-level rank confidence intervals from pairwise comparisons, and leaderboard-level rank prediction intervals using a conformal approach. This enables reliable quantification of model rank for each observed task and for new potential tasks. Experiments on simulated data and the TabArena and PromptEval (MMLU) benchmarks show that our method yields statistically valid and informative intervals, enabling reliable, uncertainty-aware model ranking on leaderboards.

Problem

• leaderboard
• rank uncertainty
• multi-task evaluation
• model ranking
• performance variability

Innovation

• rank intervals
• hierarchical framework
• conformal prediction
• uncertainty quantification
• model evaluation