Benchmark Everything Everywhere All at Once

2026-06-04

AI Summary

This work addresses the limitations of existing large model evaluation benchmarks—namely high construction costs, poor reusability, and rapid saturation—which hinder their ability to continuously differentiate the performance of state-of-the-art models. To overcome these challenges, the authors propose Benchmark Agent, the first end-to-end fully automated agent system capable of generating high-quality benchmarks with minimal human intervention. By integrating agent-based architecture, LLM-as-a-judge evaluation, human feedback, and consistency verification, the system autonomously handles query parsing, subtask design, data annotation, and quality control. It supports diverse evaluation scenarios and has successfully constructed 15 benchmarks spanning text understanding, multimodal comprehension, and domain-specific reasoning, effectively exposing models’ weaknesses in complex reasoning and demonstrating strong efficiency and scalability.

Abstract

Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance. However, their construction is labor-intensive and hard to reuse, raising concerns about sustainability and scalability. Moreover, existing benchmarks often quickly reach performance saturation after their release, resulting in insufficient discrimination among state-of-the-art models. To address these challenges, we introduce Benchmark Agent, a fully autonomous agentic system designed for benchmark building. Our framework orchestrates the complete benchmark construction pipeline, from user query analysis and subtask design to data annotation and quality control. To assess Benchmark Agent, we implement it to produce 15 representative benchmarks, spanning diverse evaluation scenarios, including text understanding, multimodal understanding, and domain-specific reasoning. Extensive experiments, including human evaluation, LLM-as-a-judge assessment, and consistency checks, demonstrate Benchmark Agent can generate high-quality benchmark samples with minimal human involvement. More importantly, through continual evaluation, we observe several insightful findings, including that current models struggle with certain domain-specific reasoning tasks. We believe that rapidly evolving benchmarks can contribute significantly to the research community. The preview and code will be publicly available at the demo page and code repository.

Problem

• benchmarks
• large language models
• multimodal models
• evaluation
• scalability

Innovation

• Benchmark Agent
• autonomous benchmark construction
• large language models
• multimodal evaluation
• continual evaluation