Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates

2024-08-23

arXiv.org

AI Summary

This work addresses critical reliability issues—bias, internal inconsistency, and prompt sensitivity—in LLM-as-a-Judge evaluation of alignment methods (e.g., RLHF, DPO). We propose the first theory-driven, interpretable, and internally consistent framework for assessing judge reliability. Methodologically, we introduce uncertainty-aware reliability metrics, systematically analyze the impact of diverse prompt templates on judge performance, and conduct empirical studies on TL;DR and HH-RLHF benchmarks. We further release an open-source visualization toolkit enabling standardized, cross-model, cross-template, and cross-task evaluation. Key findings include: (i) prompt templates substantially affect judge performance; (ii) mainstream LLM judges exhibit low agreement with human annotations; and (iii) our framework significantly enhances interpretability and reproducibility of alignment evaluation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

LLM-as-a-Judge has been widely applied to evaluate and compare different LLM alignmnet approaches (e.g., RLHF and DPO). However, concerns regarding its reliability have emerged, due to LLM judges' biases and inconsistent decision-making. Previous research has developed evaluation frameworks to assess reliability of LLM judges and their alignment with human preferences. However, the employed evaluation metrics often lack adequate explainability and fail to address LLM internal inconsistency. Additionally, existing studies inadequately explore the impact of various prompt templates when applying LLM-as-a-Judge methods, leading to potentially inconsistent comparisons between different alignment algorithms. In this work, we systematically evaluate LLM-as-a-Judge on alignment tasks by defining more theoretically interpretable evaluation metrics and explicitly mitigating LLM internal inconsistency from reliability metrics. We develop an open-source framework to evaluate, compare, and visualize the reliability and alignment of LLM judges, which facilitates practitioners to choose LLM judges for alignment tasks. In the experiments, we examine effects of diverse prompt templates on LLM-judge reliability and also demonstrate our developed framework by comparing various LLM judges on two common alignment datasets (i.e., TL;DR Summarization and HH-RLHF-Helpfulness). Our results indicate a significant impact of prompt templates on LLM judge performance, as well as a mediocre alignment level between the tested LLM judges and human evaluators.

Problem

• Assess reliability of LLM judges in alignment tasks
• Address biases and inconsistency in LLM judge decisions
• Evaluate impact of prompt templates on LLM judge performance

Innovation

• Defining interpretable metrics for LLM judges
• Mitigating LLM internal inconsistency explicitly
• Exploring diverse prompt templates impact