Problem
Research questions and friction points this paper is trying to address.
Root Cause Analysis
Nightly Test Failures
Heterogeneous Test Data
Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report
🤖 AI Summary
研究利用LLM系统辅助工业夜间测试失败的根因分析，通过单代理和多代理配置实现RCA流程，评估表明单代理系统在成本和速度上更优。
📝 Abstract
This study examines tool-augmented Large Language Model (LLM) systems for supporting Root Cause Analysis (RCA) of nightly test failures at Westermo Network Technologies AB. Nightly test executions produce heterogeneous test data and logs that practitioners currently inspect manually across multiple sources. We implemented an RCA workflow in single-agent and orchestrated multi-agent configurations, both with access to test metadata and logs. An exploratory industrial case study used two real failure scenarios. Six practitioners evaluated the scenario reports through a survey and focus group, and operational measurements were collected from 120 repeated executions. The evaluation covered practitioner-perceived correctness, reasoning quality, fix realism, clarity, usefulness, and trust, as well as cost, duration, and consistency. Neither configuration showed a consistent practitioner-perceived quality advantage across the two scenarios. The single agent system generated reports faster and at lower cost, making it the more practical baseline in this context. The potential benefits of agent architectures require further evaluation in more complex scenarios.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Large Language Model (LLM)
Root Cause Analysis (RCA)
Single-Agent
Multi-Agent
Test Metadata
🔎 Similar Papers
No similar papers found.
