When LLMs Sit Above Diagnostic Tools: Unrealized Complementarity in Industrial Fault Diagnosis

2026-10-04

AI Summary

When large language models (LLMs) serve as integration layers to fuse external diagnostic tools, information conflicts frequently cause their performance to degrade below that of standalone expert systems. Addressing industrial fault diagnosis scenarios, this study proposes an evaluation paradigm that decouples source quality from integration quality, employing paired repeated invocations, multi-dataset comparisons, and dual-source oracle methods. Results demonstrate that the accuracy of standalone experts (83.33%) significantly surpasses that of LLM-based integration (77.43%), revealing that implicit integration fails to achieve information complementarity. This work highlights the critical influence of physical evidence on robustness and provides empirical foundations for constructing ensemble frameworks that outperform their best individual components.

Abstract

Large language models are increasingly used as integration layers above specialized tools, but a stronger component does not necessarily produce a stronger combined system. Across five diagnostic datasets (bearing vibration, process monitoring, semiconductor equipment), we study whether an LLM can reliably use external diagnostic information; paired repeat calls separate advice effects from output instability. In all five, conflicting external information overturned initially correct LLM judgments. Among the four datasets with direct integration comparisons, none showed a consistent advantage for implicit LLM integration over the stronger standalone source. On a Tennessee Eastman confirmation set whose protocol was fixed before evaluation, unaided accuracy was 64.67%, implicit LLM-specialist integration 77.43%, and the specialist alone 83.33%. Specialist information improved the LLM by 12.8 points (95% interval 9.7 to 15.9), yet the integrated output stayed 5.9 points below the specialist (95% interval -12.0 to -0.7). A two-source selector oracle reached 92.76%, indicating complementarity that the integrated output did not fully realize. The integrated output missed 140 of 295 specialist corrections (47.5%) but lost 15 of 99 initially correct LLM judgments (15.2%). The deficit remained under prompt and specialist sensitivity analyses. Among CWRU cases solved under both evidence presentations, task-aligned physical evidence yielded lower estimates of susceptibility to incorrect advice in six of seven models (five intervals excluding zero); higher reasoning effort gave no reliable reduction in five models, and a separate four-model TEP analysis gave no clear evidence that it resolves the integration problem. Source quality and integration quality should be evaluated separately: an integration layer should be compared with its stronger standalone component, not only with the unaided LLM.

Problem

• Large Language Models
• Industrial Fault Diagnosis
• Tool Integration
• Complementarity
• Evaluation Methodology

Innovation

• Large Language Models
• Industrial Fault Diagnosis
• Unrealized Complementarity
• Integration Layer
• Tool-Augmented LLM