How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

2026-10-04

AI Summary

This study addresses the irreproducibility of LLM-as-a-judge evaluations caused by the absence of standardized protocols. We systematically investigate how prompt design, scoring scales, and judge model identity influence evaluation outcomes. Through comparative experiments spanning sentiment toxicity scoring and question-answering accuracy classification across multiple models, we demonstrate that design choices constitute the primary source of evaluation variance, with judge model identity playing a dominant role. Our findings confirm that while LLM judges are generally reliable, specific design configurations significantly alter their inherent biases and leniency. By identifying these critical factors, this work provides essential methodological guidance for establishing standardized and reproducible LLM evaluation pipelines.

Abstract

Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design these judges. Typically, researchers choose the prompt, rating scale, and model intuitively. If these choices change the judge's verdicts, two studies can reach different conclusions about the same facts. To address this risk and to provide an empirical basis for judge designs, we evaluate 10 reasoning models across multiple designs on two tasks: a scalar rating of sentence sentiment and toxicity (over 500 items per category), as well as a binary accuracy classification of question-answer pairs (n=600). For the rating tasks, despite judges showing significant disagreements with the human ground truth, the practical size of differences is small enough to consider most judges reliable (mean absolute deviation of 0.11 points on a 1 - 7 scale); toxicity judges even outperform standard classifiers. Judges are also highly accurate on average (96.5%) for the accuracy classification task. However, design choices can produce shifts: changing the rating scale alone can shift measured bias by up to 0.93 points (rating task), and while accuracy levels are rarely impacted, design choices consistently impact judge leniency (classification task; leniency drop of 28.9 percentage points when using detailed prompts, and up to 56.1 percentage points when switching models). Counterintuitively, lower reasoning effort affects neither accuracy nor leniency. Across both tasks, model identity is the dominant source of variance. These findings suggest that while LLM judges are broadly trustworthy in aggregate, design choices can be meaningful sources of variance. Given the growing reliance on automated evaluation in LLM research, we intend this study as a methodological reference for designing more robust and replicable LLM-as-a-judge pipelines.

Problem

• LLM-as-a-judge
• prompt design
• rating scales
• evaluation reliability
• automated evaluation

Innovation

• LLM-as-a-Judge
• Prompt Design
• Rating Scales
• Automated Evaluation
• Model Variance