Dynamically Allocating Evaluation Effort for Model Ranking

2026-08-04

AI Summary

This work addresses the high cost and inefficiency of human evaluation in traditional NLP model assessment, particularly the redundancy incurred when distinguishing among top-performing models. It reframes multi-model evaluation as an optimal arm identification problem in a multi-armed bandit setting with correlated arms and introduces a theoretically optimal adaptive sampling strategy. By dynamically allocating annotation resources to focus on the most competitive models, the proposed method integrates correlation modeling with statistical decision theory. This approach significantly reduces the required amount of human evaluation while maintaining rigorous guarantees for correctly identifying the best model, thereby substantially improving both evaluation efficiency and discriminative power—making it especially well-suited for large-scale model competitions.

Abstract

While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability. When identifying top-performing models, typical evaluation protocols waste effort by exhaustively evaluating all models on the entire benchmark, a safe but inefficient approach. In this work, we formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms, where pulling an arm corresponds to human-evaluating a model. By sampling adaptively based on the intermediate model rankings obtained on the samples so far, we can focus the annotation budget on the most competitive models. We prove the optimality of the proposed algorithms and show that it improves discrimination between top-performing models. This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.

Problem

• human evaluation
• model ranking
• evaluation efficiency
• best-arm identification
• NLP benchmarks

Innovation

• best-arm identification
• multi-armed bandit
• adaptive sampling
• human evaluation
• model ranking