AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility

2026-06-11

AI Summary

Current agent evaluation lacks open, general, and reproducible interfaces, leading to a disconnect between benchmarking and real-world deployment. This work proposes the Agentic Agent Assessment (AAA) framework, which introduces evaluator agents to conduct assessments solely through a unified Agent-to-Agent (A2A) task protocol and Model Calling Protocol (MCP), enabling interoperable, reproducible, and multi-agent collaborative evaluation of heterogeneous agents. The framework supports five operational modes that balance openness, privacy, and practicality. Its advantages in evaluation breadth, utility, and fidelity are demonstrated through a five-month open competition involving 298 evaluator agents and 467 participant agents, as well as a case study on programming agents.

Abstract

Agent systems are advancing quickly across domains, but their evaluation remains fragmented. Most benchmarks rely on fixed, LLM-centric harnesses that require heavy integration, create test-production mismatch, and limit fair comparison across diverse agent designs. The root problem is the lack of an open, agent-agnostic assessment interface. We advocate Agentified Agent Assessment (AAA), where evaluation is performed by judge agents and all participants interact through standardized protocols: A2A for task management and MCP for tool access. Conventional benchmarking defines two separate interfaces, one for the benchmark and one for the agent, while AAA only needs one; this yields a generic, unified framework that separates assessment logic from agent implementation and enables reproducible, interoperable, and multi-agent evaluation. We further introduce AgentBeats as a concrete realization of AAA: we identify five practical operation modes that make standardized assessment compatible with real-world constraints on openness, privacy, and reproducibility. To evaluate our design at scale, we conduct two studies: a five-month open competition that drew 298 judge agents across 12 categories together with 467 subject agents from independent participants, showing that AAA applies across a heterogeneous range of benchmarks; and a case study on coding agents that confirms agentified evaluation preserves fidelity with the public record while surfacing previously missing head-to-head results, yielding research insights about agent design. Combining a community-scale field study and a controlled coding case study, we verify that AAA delivers coverage, practicality, and fidelity across heterogeneous scenarios at scale. Together, AAA and AgentBeats offer a clear path toward open, standardized, and reproducible agent assessment.

Problem

• agent evaluation
• benchmarking
• standardization
• reproducibility
• openness

Innovation

• Agentified Agent Assessment
• standardized evaluation
• judge agents
• A2A protocol
• MCP protocol