Approximating Human Preferences Using a Multi-Judge Learned System

2025-10-29

AI Summary

This paper addresses the challenge of aligning LLM-based evaluators with human preferences—characterized by miscalibration, sensitivity to scoring criteria, inherent biases, and output instability—by proposing a multi-judge learning framework. Methodologically, it introduces (1) a role-aware preference labeling approach that explicitly models the diversity of human judgments; (2) a unified aggregation framework for multi-judge outputs, incorporating dual-path aggregators—Generalized Additive Models (GAM) and Multi-Layer Perceptrons (MLP)—to enhance robustness; and (3) a score-conditioned judging mechanism that decouples absolute scoring from relative preference estimation. Empirically, the framework achieves significant improvements over baselines in reward modeling and model routing tasks. It demonstrates superior resilience to judge bias and output instability, yielding preference distributions closer to ground-truth human judgments. This work establishes a novel paradigm for trustworthy LLM evaluation.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Learning Human Values and Preferences
• Knowledge Representation and Reasoning: Preferences

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Aligning LLM-based judges with human preferences is a significant challenge, as they are difficult to calibrate and often suffer from rubric sensitivity, bias, and instability. Overcoming this challenge advances key applications, such as creating reliable reward models for Reinforcement Learning from Human Feedback (RLHF) and building effective routing systems that select the best-suited model for a given user query. In this work, we propose a framework for modeling diverse, persona-based preferences by learning to aggregate outputs from multiple rubric-conditioned judges. We investigate the performance of this approach against naive baselines and assess its robustness through case studies on both human and LLM-judges biases. Our primary contributions include a persona-based method for synthesizing preference labels at scale and two distinct implementations of our aggregator: Generalized Additive Model (GAM) and a Multi-Layer Perceptron (MLP).

Problem

• Modeling diverse human preferences using multi-judge aggregation
• Addressing rubric sensitivity and bias in LLM-based judges
• Creating scalable persona-based preference synthesis methods

Innovation

• Aggregating multiple rubric-conditioned judges outputs
• Using persona-based method for scaling preference labels
• Implementing GAM and MLP as preference aggregators