An End-to-End Agent Auditing Engine

2026-08-07

AI Summary

This work addresses the lack of an efficient, systematic, and comprehensive evaluation framework for modern AI agents, which struggles to keep pace with rapidly evolving deployment infrastructures. To this end, we propose A²E, an end-to-end agent auditing engine that introduces the novel Agent Task Protocol to decouple tasks from underlying frameworks, enabling unified integration of diverse evaluation scenarios. Leveraging automated instrumentation and standardized execution traces, A²E establishes a multidimensional assessment methodology encompassing execution efficiency, tool utilization, task planning, and error recovery. Experimental results reveal substantial performance variations across different model–framework combinations on various tasks, with no single configuration consistently dominating others. These findings underscore the necessity of systematic benchmarking and provide empirical grounding for the co-optimization of models and agent frameworks.

Abstract

With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains. The fast-evolving harness ecosystem has also made rigorous capability evaluation increasingly important. However, efficiently building an end-to-end, systematic, and comprehensive evaluation pipeline remains a significant challenge. To address this challenge, we introduce $A^2E$ (Agent Auditing Engine), an end-to-end evaluation engine designed for agent harnesses. $A^2E$ leverages our newly proposed Agent Task Protocol (ATP) to enable the rapid integration of evaluation tasks with different harnesses. Through an automatically instrumented Monitor, it captures and generates standardized execution traces during experiments. In the Evaluation stage, $A^2E$ systematically assesses harness capabilities using a suite of multidimensional metrics. Compared with correctness alone, these metrics provide a more fine-grained characterization of differences among harnesses in execution efficiency, tool use, task planning, and error recovery. Experiments conducted with $A^2E$ further reveal that model-harness combinations exhibit substantial performance variation across different types of tasks, and that no single combination consistently outperforms all others across every task. These findings not only demonstrate the necessity of systematic evaluation but also provide useful guidance for the co-evolving of models and harnesses. Our code is available at https://github.com/datamllab/A2E.

Problem

• agent evaluation
• large language models
• harness ecosystem
• systematic assessment
• execution traces

Innovation

• Agent Auditing Engine
• Agent Task Protocol
• End-to-End Evaluation
• Execution Traces
• Multidimensional Metrics