🤖 AI Summary
This work addresses the limitations of root cause analysis (RCA) in real-world microservice systems, where both classical methods and large language models (LLMs) struggle due to the lack of effective reasoning mechanisms over large-scale, multimodal telemetry data—encompassing metrics, logs, and traces. The authors propose a structured multi-agent RCA framework that operates with or without domain knowledge and introduces a novel backward-reasoning agent to distinguish between “reasoning gaps” and “data ambiguity,” revealing that performance bottlenecks stem from insufficient model reasoning capabilities rather than data scarcity. The framework further integrates automated rule mining to reduce reliance on manual expert knowledge. Experimental results on the OpenRCA benchmark demonstrate that the proposed approach significantly outperforms existing LLM-based and classical RCA methods, underscoring the critical importance of enhanced reasoning and the synergistic integration of domain knowledge.
📝 Abstract
Identifying root causes in production microservice failures requires reasoning over large-scale, multimodal telemetry spanning metrics, logs, and traces, a problem that has proved resistant to both classical and LLM-based approaches. The OpenRCA dataset exemplifies these challenges: it is large-scale, multimodal, and lacks detailed domain knowledge, and yields consistently low accuracy across all existing methods. We show that classical causal discovery methods and existing LLM-based multi-agent systems fail to reliably identify root causes on this benchmark, and present a Structured Multi-Agent RCA pipeline that substantially outperforms existing LLM-based and classical baselines, supporting both domain-knowledge and knowledge-free operating modes. To diagnose where failures originate, we introduce a reverse reasoning agent that, given the correct answer, identifies which signals in the extracted anomalies support it and determines whether Stage~1 had access to those signals, classifying each failure as Reasoning Gap (evidence present but unused) or Data Ambiguity (evidence genuinely absent). This analysis reveals that the required evidence is present in the vast majority of failures: the bottleneck is not data access but the agent's ability to reason over it correctly. We further introduce an automated rule mining pipeline that systematically extracts discrimination rules from reverse reasoning reports, reducing reliance on manual knowledge curation. Across all configurations, model reasoning capability and domain knowledge are the primary constraints: stronger models embed more domain expertise, and explicit knowledge injection partially compensates for this gap. Reasoning performance remains practically bounded even when evidence extraction is perfect: scaffold engineering and better data pipelines alone cannot close this gap; progress requires improvements at the model level.