orchestration benchmarking

This work addresses the limitation of existing benchmarks, which focus solely on accuracy in multi-agent orchestration tasks while neglecting fine-grained diagnosis of failure origins and recovery capabilities. The authors propose a reproducible fault-injection framework to systematically evaluate failure modes, task decomposition quality, and recovery mechanisms within templated enterprise workflows. They introduce two novel metrics: “cascade radius” and failure-mode-specific recovery rates, and employ controlled probes to analyze recovery behavior across different fault types. Experimental results demonstrate that intent-based reasoning routing achieves 100% recovery under adversarial conditions, significantly outperforming keyword-based routing; tool-related failures are fully recoverable, whereas semantic failures prove largely irrecoverable; and cascade radius increases with workflow depth.

Existing evaluation methods struggle to disentangle the quality of task orchestration in multi-agent systems from confounding factors such as agent capabilities and environmental noise, while real-world execution incurs prohibitive costs. To address this, this work proposes OrchBench—a deterministic simulation-based benchmarking platform that models task dependencies via directed acyclic graphs and enables isolated, efficient assessment of orchestration plans. OrchBench achieves the first interpretable evaluation of orchestration quality with dramatically reduced overhead: requiring only 1.3% of the tokens and 10.3% of the time compared to real execution, while maintaining high fidelity (Pearson r = 0.816). Furthermore, it reveals that information retention rate is more critical to performance than simply increasing the number of agents.

Enterprise-grade multi-agent systems lack empirical studies on cross-architectural interactions; existing work typically evaluates components in isolation, obscuring the true efficacy of configuration combinations. Method: We introduce the first enterprise-oriented agent architecture benchmark, systematically evaluating 18 configurations across four core dimensions—orchestration strategy, prompt engineering, memory architecture, and tool integration—using ReAct versus function-calling paradigms, multi-dimensional ablation analysis, and realistic task frameworks. Contribution/Results: We identify strong architectural preferences, challenging the “one-size-fits-all” design paradigm; the best-performing configuration achieves only 35.3% and 70.8% success rates on complex and simple tasks, respectively, revealing fundamental performance bottlenecks. Our findings advance the development of enterprise-tailored agent architectures grounded in empirical evidence and systematic evaluation.

Existing benchmarks struggle to evaluate the impact of harness components on the performance of large language model (LLM) agent systems, often overlooking execution details or fixing harness configurations. This work proposes Harness-Bench, the first framework to systematically decouple and quantify how harness design influences agent workflows. By enforcing a unified task environment, computational budget, and evaluation protocol—and leveraging sandboxed offline tasks, realistic usage patterns, human auditing, and full execution trace logging—it enables controlled experiments across diverse model–harness combinations. Analysis of 5,194 execution traces across 106 tasks reveals significant differences in completion rates, process quality, efficiency, and failure modes, exposing alignment failures stemming from misalignment between reasoning and execution. The study argues that agent performance should be reported based on joint model–harness configurations rather than base models alone.

This work addresses the limitations of existing enterprise-scale multi-agent systems, which predominantly rely on discrete request-response paradigms and struggle to support large-scale, continuous event-driven monitoring and response. To overcome this, we propose an autonomous, event-driven multi-agent collaboration framework tailored for enterprise AI. Our approach presents the first systematic evaluation at enterprise scale of DAG-based Plan-and-Execute and ReAct architectures, augmented with a task manager that enables priority-aware reasoning, correlated event aggregation, and preemptive scheduling. Experimental results across Persona–Department–Enterprise tiered scenarios demonstrate that our method reduces latency for high-priority tasks by 14%–75% and improves accuracy in handling correlated events by over 20 percentage points. These findings indicate that system bottlenecks stem primarily from scale rather than intrinsic task complexity.

This work addresses the challenges faced by large language model (LLM) agents operating over flat tool registries—namely, combinatorial explosion in decision space, context saturation, and degraded routing accuracy. To overcome these limitations, the authors propose a skill-tree-based hierarchical architecture that separates routing logic at internal nodes from execution at leaf nodes. Inspired by pushdown automata, the framework incorporates a LIFO stack-frame memory model and a lazy capability discovery mechanism, enabling isolated execution paths and scalable context management. The approach supports manifest-driven single-step execution loops and formal state modeling, significantly improving routing accuracy while reducing memory footprint and prompt costs under conditions of tool proliferation, multi-step workflows, and prompt exposure. This design meets enterprise-grade requirements for isolation and scalability.

This work addresses the orchestration bottlenecks faced by ultra-large-scale Sim-AI workflows on leadership-class supercomputers, which arise from task heterogeneity and extreme ensemble sizes. To overcome these challenges, the authors propose EnsembleLauncher, a recursively hierarchical and fully decentralized workflow orchestrator that introduces a decentralized control plane and a programmable scheduling policy interface, thereby surpassing conventional tools in both scalability and scheduling flexibility. Experiments on the Aurora supercomputer demonstrate that EnsembleLauncher can efficiently schedule system-wide resources to support up to 8 million serial tasks, achieving more than a fourfold performance improvement over state-of-the-art alternatives. Furthermore, it significantly enhances resource utilization for workloads with high task variance and active learning pipelines.

This work addresses the limitations of traditional high-performance computing (HPC), which relies on manual task scripting and scheduling and struggles to meet the automation demands of complex scientific workflows. The authors propose the first large language model–based autonomous agent framework that enables end-to-end automated execution of HPC workflows from descriptive instructions. The framework integrates Slurm/Flux job schedulers, low-latency AWS cloud infrastructure, and event monitoring mechanisms to support task definition, optimization, and scheduling. Experimental results demonstrate that the system efficiently deploys scalable experiments, accurately translates job specifications—with only occasional deviations in processor affinity—and successfully reproduces an expert-level variant calling pipeline, achieving consistent results in 18 out of 19 runs. These findings validate the framework’s feasibility and effectiveness in real-world HPC environments.