Evaluating Scoring Bias in LLM-as-a-Judge

2025-06-27

AI Summary

This work addresses scoring bias in the LLM-as-a-Judge paradigm, introducing— for the first time—the formal definition of bias in scoring tasks and a systematic evaluation framework. Methodologically, it proposes perturbation-robustness-driven multidimensional stability metrics, covering three critical variables: prompt templates, scoring criteria, and reference answers; integrates data synthesis to augment benchmarks; and conducts controlled-variable experiments across mainstream scoring LLMs. Results reveal pervasive scoring instability in current LLM judges, with significant sensitivity to prompt engineering and reference answer selection. The study uncovers underlying bias generation mechanisms and empirically validates that optimizing prompt structure and standardizing reference answers effectively mitigate bias. This work establishes a novel, principled paradigm for trustworthy AI evaluation and delivers a reproducible, open toolchain for bias assessment in LLM-based scoring systems.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Ethics — Bias, Fairness, Transparency & PrivacyPhilosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

Search and Retrieval-Augmented AI: Web evaluation methodologies and metricsEconomics, Online Markets and Human Computation: LLM based quality controls for crowd workUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The remarkable performance of Large Language Models (LLMs) gives rise to``LLM-as-a-Judge'', where LLMs are employed as evaluators for complex tasks. Moreover, it has been widely adopted across fields such as Natural Language Processing (NLP), preference learning, and various specific domains. However, there are various biases within LLM-as-a-Judge, which adversely affect the fairness and reliability of judgments. Current research on evaluating or mitigating bias in LLM-as-a-Judge predominantly focuses on comparison-based evaluations, while systematic investigations into bias in scoring-based evaluations remain limited. Therefore, we define scoring bias in LLM-as-a-Judge as the scores differ when scoring judge models are bias-related perturbed, and provide a well-designed framework to comprehensively evaluate scoring bias. We augment existing LLM-as-a-Judge benchmarks through data synthesis to construct our evaluation dataset and design multi-faceted evaluation metrics. Our experimental results demonstrate that the scoring stability of existing judge models is disrupted by scoring biases. Further exploratory experiments and discussions provide valuable insights into the design of scoring prompt templates and the mitigation of scoring biases on aspects such as score rubrics, score IDs, and reference answer selection.

Problem

• Evaluating scoring bias in LLM-as-a-Judge systems
• Assessing fairness and reliability of LLM judgments
• Investigating bias mitigation in scoring-based evaluations

Innovation

• Framework evaluates scoring bias in LLMs
• Data synthesis augments benchmark datasets
• Multi-faceted metrics assess scoring stability