Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge

2026-05-28

AI Summary

This work addresses the scalability and cost limitations of existing LLM-as-a-Judge approaches, which rely on human-annotated reference answers or expert-defined scoring rubrics. The authors propose a training-free, fully annotation-free method that dynamically generates scoring criteria at both dataset-level and instance-level granularity. A meta-evaluation mechanism drives iterative refinement to continuously improve the scoring criterion generation model. Experimental results demonstrate that the proposed approach matches or exceeds state-of-the-art methods across four benchmarks. Notably, a fine-tuned 14B open-source model outperforms all baselines in both pairwise and pointwise evaluation settings, even surpassing larger closed-source models.

Abstract

LLM-as-a-Judge is a scalable alternative to human evaluation, yet existing rubric-based methods rely on human-annotated data such as reference answers or expert-crafted rubrics. We propose to automatically generate fine-grained evaluation rubrics without any human annotation. Our training-free method generates rubrics at dataset-specific and instance-specific granularities, achieving performance competitive with existing methods across four benchmarks. We further present a method that iteratively fine-tunes a rubric generator model via meta-judge reward signals. The fine-tuned generator outperforms all existing baselines in both pairwise and pointwise evaluation. Notably, a fine-tuned 14B rubric generator outperforms a much larger proprietary model at rubric generation, showing the effectiveness of our fine-tuning strategy.

Problem

• LLM-as-a-Judge
• evaluation rubrics
• human annotation
• automatic rubric generation
• dynamic rubrics

Innovation

• LLM-as-a-Judge
• automatic rubric generation
• training-free evaluation
• meta-judge reward
• iterative fine-tuning