Diagnosing Failure Root Causes in Platform-Orchestrated Agentic Systems: Dataset, Taxonomy, and Benchmark

2025-09-28

AI Summary

Systematic root-cause diagnosis for failures in platformized multi-agent systems remains underexplored. Method: We introduce AgentFail—a first-of-its-kind, fine-grained annotated failure log dataset (307 samples)—and propose the first taxonomy for failure root-cause classification in this domain. Leveraging this taxonomy, we design a classification-guided large language model prompting framework that integrates counterfactual reasoning and human verification to ensure annotation reliability, and release a reproducible automated diagnosis benchmark. Results: Experiments reveal that state-of-the-art methods achieve only 33.6% accuracy, underscoring the task’s substantial difficulty. Our work provides empirical evidence and practical guidelines for enhancing the robustness of multi-agent system design, establishing foundational resources and evaluation protocols for future research.

Technology Category

• Multiagent Systems: Multiagent Planning
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Agentic systems consisting of multiple LLM-driven agents coordinating through tools and structured interactions, are increasingly deployed for complex reasoning and problem-solving tasks. At the same time, emerging low-code and template-based agent development platforms (e.g., Dify) enable users to rapidly build and orchestrate agentic systems, which we refer to as platform-orchestrated agentic systems. However, these systems are also fragile and it remains unclear how to systematically identify their potential failure root cause. This paper presents a study of root cause identification of these platform-orchestrated agentic systems. To support this initiative, we construct a dataset AgentFail containing 307 failure logs from ten agentic systems, each with fine-grained annotations linking failures to their root causes. We additionally utilize counterfactual reasoning-based repair strategy to ensure the reliability of the annotation. Building on the dataset, we develop a taxonomy that characterizes failure root causes and analyze their distribution across different platforms and task domains. Furthermore, we introduce a benchmark that leverages LLMs for automatically identifying root causes, in which we also utilize the proposed taxonomy as guidance for LLMs. Results show that the taxonomy can largely improve the performance, thereby confirming its utility. Nevertheless, the accuracy of root cause identification reaches at most 33.6%, which indicates that this task still remains challenging. In light of these results, we also provide actionable guidelines for building such agentic systems. In summary, this paper provides a reliable dataset of failure root cause for platform-orchestrated agentic systems, corresponding taxonomy and benchmark, which serves as a foundation for advancing the development of more reliable agentic systems.

Problem

• Identifying failure root causes in platform-orchestrated multi-agent systems
• Developing taxonomy and benchmark for agentic system failure analysis
• Addressing fragility of LLM-driven agent coordination systems

Innovation

• Constructed AgentFail dataset with annotated failure logs
• Developed taxonomy to characterize failure root causes
• Introduced benchmark using LLMs for root cause identification