Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons

2025-06-04

AI Summary

Existing LLM evaluation methods rely predominantly on single-round pairwise comparisons or independent scoring, limiting their ability to establish globally consistent and robust rankings. Method: This paper introduces an iterative pairwise comparison framework inspired by sports knockout tournaments—the first such application in LLM evaluation. It employs multi-round dynamic adversarial comparisons to accumulate preference signals and calibrate judgments via feedback, while incorporating explicit consistency constraints to enhance ranking robustness. The approach integrates preference modeling, rank aggregation, and iterative optimization. Results: Evaluated on university exam grading and machine translation assessment tasks, the method increases the average Pearson correlation between LLM scores and expert judgments by 0.07, significantly improving alignment with human preferences. It establishes a novel paradigm for deploying LLMs as reliable, trustworthy evaluators.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: (Large) Language Models
• Humans and AI: Learning Human Values and Preferences

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have shown to be effective evaluators across various domains such as machine translations or the scientific domain. Current LLM-as-a-Judge approaches rely mostly on individual assessments or a single round of pairwise assessments, preventing the judge LLM from developing a global ranking perspective. To address this, we present Knockout Assessment, an LLM-asa Judge method using a knockout tournament system with iterative pairwise comparisons. Experiments across three LLMs on two datasets show that knockout assessment improves scoring accuracy, increasing Pearson correlation with expert evaluations by 0.07 on average for university-level exam scoring and machine translation evaluations, aligning LLM assessments more closely with human scoring.

Problem

• Improving LLM evaluation accuracy through iterative pairwise comparisons
• Addressing limitations of single-round LLM-as-a-Judge assessments
• Aligning LLM scoring more closely with human expert evaluations

Innovation

• Uses knockout tournament system for evaluations
• Implements iterative pairwise comparisons method
• Improves scoring accuracy with human alignment