CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks

2026-06-02

AI Summary

This work addresses the challenge of reliable model selection in scenarios lacking task-specific annotations and where existing benchmarks are compromised by pretraining data contamination. The authors propose CoEval, a framework that leverages only a task description to enable a teacher model to dynamically generate novel, uncontaminated evaluation sets with controllable attributes. An ensemble of cross-vendor large language models acts as automated judges to produce end-to-end model rankings without human intervention. CoEval establishes the first automatic evaluation paradigm that operates without ground-truth labels or trusted benchmarks, employing 13-gram deduplication to ensure zero overlap with mainstream benchmarks, thereby mitigating data leakage and source bias. Experiments demonstrate that CoEval accurately reproduces ground-truth rankings on labeled tasks (ρ = 0.86), with a cost of only $5.89 for 7,978 evaluations across four tasks in a single run, highlighting the critical role of judge diversity in assessment reliability.

Abstract

Choosing or ranking language models for a specific application is hardest when no task-specific labeled data exists, and standard public benchmarks cannot be trusted, their items having likely leaked into pretraining, so scores reflect memorization rather than fitness. We present CoEval, an open-source, reusable framework that closes this gap end to end: from only a description of a task or domain, teacher models synthesize a fresh, attribute-controlled benchmark with no human labels, contamination-free because items are generated anew on each run, and a cross-family judge ensemble ranks candidate models with no human raters. Validated where ground truth exists, CoEval recovers the true model ranking and tracks ground-truth correctness at ho=0.86. The label-free judging needs no human calibration because judge-panel composition (vendor diversity), not size, drives reliability: a small, well-chosen cross-family panel is most reliable, while a single judge can be anti-correlated with ground truth (judge-choice regret 0.35) and the ensemble never is. Generated items show zero verbatim 13-gram overlap with five major public benchmarks; the panel cancels verbosity bias and precludes same-family self-preference. A four-task study produced 7,978 evaluations for USD 5.89. The same declarative pipeline applies to any domain and is cheap enough to re-run on every model release: a label-free, contamination-free leaderboard any team can regenerate for its own application.

Problem

• language model ranking
• unlabeled data
• benchmark contamination
• custom tasks
• model evaluation

Innovation

• label-free evaluation
• contamination-free benchmarking
• cross-family judge ensemble
• synthetic benchmark generation
• model ranking without human labels