agentic system benchmarking

Current LLM agent evaluation lacks a systematic framework, particularly neglecting enterprise-specific requirements such as role-based access control, regulatory compliance, and long-horizon interactive behavior. To address this gap, we conduct a comprehensive literature review and propose the first two-dimensional evaluation taxonomy: one axis captures *objective dimensions*—including behavior, capability, reliability, and security—while the other captures *process dimensions*—encompassing interaction paradigms, benchmark datasets, evaluation metrics, and toolchains. Crucially, our taxonomy explicitly incorporates enterprise challenges, exposing critical shortcomings in existing work regarding holistic coverage, scalability, and real-world applicability. The framework provides researchers and practitioners with a structured, actionable reference for designing, evaluating, and deploying LLM agents in complex, mission-critical environments. It advances the field toward trustworthy, production-ready LLM agent systems. (138 words)

Current evaluations of large language model (LLM) agents are often confounded by implementation-specific details and environmental variability, hindering fair assessment of intrinsic model capabilities. This work proposes a unified evaluation framework that standardizes diverse benchmarks into a consistent instruction–tool–environment format, executed within a controlled sandbox under a fixed ReAct architecture and supported by offline snapshots to decouple environmental influences. For the first time, cross-benchmark standardized evaluation is achieved, introducing unified metrics for resource consumption and a failure attribution taxonomy distinguishing decision-making from execution errors. The framework integrates seven major benchmarks spanning 24 domains, encompassing 400,000 rollouts and 5 billion tokens, revealing significant impacts of architectural and environmental factors on performance and effectively isolating true model capabilities from external interference.

This work addresses critical challenges in the real-world deployment of large language model (LLM)-driven agent systems, particularly concerning robustness, safety, and reliability. Bridging academic advances with industrial practice, the study presents case studies from software engineering, scientific discovery, and finance to distill reusable design patterns and an evaluation checklist. It integrates key techniques including LLM-based reasoning and planning, multi-agent coordination, validation pipelines, fallback mechanisms, and human-in-the-loop oversight. The proposed cross-domain deployment framework has been validated in pharmaceutical discovery and financial systems, demonstrating significant improvements in stability and trustworthiness of agent systems in real-world settings, thereby narrowing the gap between research innovation and practical implementation.

This work addresses the absence of standardized evaluation methodologies for large language model–based multi-agent systems (MAS) in terms of testing, reliability, and observability. We propose a unified execution framework that enables seamless integration of both native and third-party MAS through lightweight adapters, standardized interfaces, and a cross-framework example library, while capturing framework-agnostic execution traces and system-level metrics such as latency, cost, and failure rates. Our systematic evaluation—the first of its kind—demonstrates that MAS architectural design predominantly governs runtime stability, performance variability, and the trade-offs among cost, latency, and accuracy, with effects far outweighing those of backend model choices or tool configurations. These findings are empirically validated across twelve representative MAS implementations.

This work addresses the critical yet underexplored impact of architectural design on the performance of multi-agent large language model (LLM) frameworks, where a lack of standardized evaluation methodologies has hindered systematic comparison. To bridge this gap, we propose the first comprehensive architectural taxonomy for multi-agent LLM systems and introduce MAFBench, a unified benchmark that enables controlled cross-framework evaluation through standardized execution protocols. Our experiments reveal that architectural choices can lead to over 100-fold increases in latency, up to 30% degradation in planning accuracy, and a dramatic drop in collaboration success rates—from above 90% to below 30%. Based on these findings, we derive practical architectural design principles and framework selection guidelines to inform real-world deployment.

Enterprise-grade multi-agent systems lack empirical studies on cross-architectural interactions; existing work typically evaluates components in isolation, obscuring the true efficacy of configuration combinations. Method: We introduce the first enterprise-oriented agent architecture benchmark, systematically evaluating 18 configurations across four core dimensions—orchestration strategy, prompt engineering, memory architecture, and tool integration—using ReAct versus function-calling paradigms, multi-dimensional ablation analysis, and realistic task frameworks. Contribution/Results: We identify strong architectural preferences, challenging the “one-size-fits-all” design paradigm; the best-performing configuration achieves only 35.3% and 70.8% success rates on complex and simple tasks, respectively, revealing fundamental performance bottlenecks. Our findings advance the development of enterprise-tailored agent architectures grounded in empirical evidence and systematic evaluation.

This study addresses the absence of systematic design principles for scalable multi-agent systems. The authors propose four core design principles oriented toward scalability, formulate a reference architecture based on constrained directed workflow graphs, and introduce summarization-based communication, elastic feedback, and sequential coordination mechanisms. By evaluating configurations of varying complexity on standardized end-to-end tasks, the work formalizes workflow topology for the first time and reveals the critical influence of large language model (LLM) capability thresholds on system scalability. Experimental results demonstrate that, provided LLM capabilities meet a minimum threshold, system scaling yields improved accuracy with near-linear cost growth; however, excessive architectural complexity leads to performance degradation, and consistency challenges persist across all levels of scale.

Current large language model–driven multi-agent systems suffer from a lack of formal specification, verifiability, and production-grade reliability, leading to poor reproducibility and significant challenges in transitioning from experimentation to deployment. This work proposes MAS-Lab, a novel framework that introduces a three-layer architecture—comprising a specification layer, an operating system layer, and an experimentation layer—to decouple semantic intent from execution logic. By enabling declarative specification–based development and verification, MAS-Lab integrates a stateful multi-agent operating system (MAS-OS) alongside observability and evaluation tooling, thereby unifying intent-driven verification, evolution, and deployment. The approach substantially enhances system reliability, maintainability, and engineering efficiency, offering a comprehensive methodology for the industrial-scale realization of multi-agent systems.

Existing benchmarks for microservice fault diagnosis focus solely on final answers, overlooking the systematic reasoning processes of large language model agents. This work proposes the first evaluation paradigm centered on the diagnostic reasoning process, introducing AIOps2025 and RCA100—large-scale, expert-annotated datasets encompassing three key dimensions: fault localization, identification, and root cause attribution. These datasets integrate multimodal observability data with causal reasoning analysis and cover over 500 real-world failure cases. The benchmark’s effectiveness has been validated through an international competition involving more than 6,000 teams, establishing it as the first reasoning-oriented benchmark for intelligent microservice fault diagnosis to be empirically validated at scale.

This work addresses the frequent failure of research code deployment due to complex environment configurations, heterogeneous toolchains, system dependencies (e.g., GPU/CUDA), and legacy compatibility issues—challenges inadequately captured by existing benchmarks. To bridge this gap, we introduce DeployBench, the first systematic, multidimensional deployment benchmark encompassing 51 research artifacts across AI/ML, computer systems, and scientific computing. DeployBench evaluates the autonomous deployment capabilities of LLM agents through hidden validation pipelines that reproduce experiments and verify outputs. Built upon the OpenHands framework, our evaluation integrates four state-of-the-art LLMs and executes end-to-end deployment and validation in full system environments. Results reveal that even the best-performing agent achieves a success rate of only 7.8%–51.0%, with 63% of failures attributed to premature self-termination or misaligned validation objectives, exposing critical deficiencies in agents’ judgment of task completion.