LLMEval-3: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models

2025-08-07

AI Summary

Existing LLM evaluations rely heavily on static benchmarks, rendering them vulnerable to data contamination and leaderboard overfitting, thus failing to reflect models’ true capabilities. Method: We propose the first dynamic evaluation framework designed for long-term evolution: (1) a 220K-item graduate-level question bank enabling dynamic sampling of unseen test sets per run; (2) an anti-cheating model architecture, contamination-resistant data curation, and a calibrated LLM-as-a-judge system achieving 90% inter-annotator agreement; and (3) relative ranking to mitigate absolute scoring bias. Contribution/Results: Over 20 months, we evaluated nearly 50 mainstream models, uncovering—for the first time in longitudinal assessment—the knowledge retention bottleneck and contamination blind spots in LLMs. Our framework significantly improves ranking stability and evaluation reliability, surpassing the performance ceiling identification limits inherent to static benchmarks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Existing evaluation of Large Language Models (LLMs) on static benchmarks is vulnerable to data contamination and leaderboard overfitting, critical issues that obscure true model capabilities. To address this, we introduce LLMEval-3, a framework for dynamic evaluation of LLMs. LLMEval-3 is built on a proprietary bank of 220k graduate-level questions, from which it dynamically samples unseen test sets for each evaluation run. Its automated pipeline ensures integrity via contamination-resistant data curation, a novel anti-cheating architecture, and a calibrated LLM-as-a-judge process achieving 90% agreement with human experts, complemented by a relative ranking system for fair comparison. An 20-month longitudinal study of nearly 50 leading models reveals a performance ceiling on knowledge memorization and exposes data contamination vulnerabilities undetectable by static benchmarks. The framework demonstrates exceptional robustness in ranking stability and consistency, providing strong empirical validation for the dynamic evaluation paradigm. LLMEval-3 offers a robust and credible methodology for assessing the true capabilities of LLMs beyond leaderboard scores, promoting the development of more trustworthy evaluation standards.

Problem

• Addresses data contamination in static LLM evaluations
• Introduces dynamic sampling for fair model comparison
• Ensures evaluation integrity via anti-cheating measures

Innovation

• Dynamic sampling of unseen test sets
• Contamination-resistant data curation
• Calibrated LLM-as-a-judge process