🤖 AI Summary
This paper addresses the under-recognized reliability challenges of large language models (LLMs) in real-world system deployments. Adopting a systems engineering perspective, it establishes the first fault taxonomy for LLM-based applications. Through systematic analysis and multi-case root-cause investigation, the study identifies 15 classes of latent failures—including multi-step reasoning drift, context boundary degradation, erroneous tool invocation, and latent inconsistency—exposing fundamental limitations of current evaluation benchmarks in stability, reproducibility, and workflow integration. The work introduces high-level design principles centered on observability, cost sensitivity, and version evolution, shifting LLM reliability research from a model-centric to a system-integration paradigm. It delivers the first structured fault classification framework and practical guidance for building reliable, maintainable, and auditable LLM software systems.
📝 Abstract
Large language models (LLMs) are being rapidly integrated into decision-support tools, automation workflows, and AI-enabled software systems. However, their behavior in production environments remains poorly understood, and their failure patterns differ fundamentally from those of traditional machine learning models. This paper presents a system-level taxonomy of fifteen hidden failure modes that arise in real-world LLM applications, including multi-step reasoning drift, latent inconsistency, context-boundary degradation, incorrect tool invocation, version drift, and cost-driven performance collapse. Using this taxonomy, we analyze the growing gap in evaluation and monitoring practices: existing benchmarks measure knowledge or reasoning but provide little insight into stability, reproducibility, drift, or workflow integration. We further examine the production challenges associated with deploying LLMs - including observability limitations, cost constraints, and update-induced regressions - and outline high-level design principles for building reliable, maintainable, and cost-aware LLM systems. Finally, we outline high-level design principles for building reliable, maintainable, and cost-aware LLM-based systems. By framing LLM reliability as a system-engineering problem rather than a purely model-centric one, this work provides an analytical foundation for future research on evaluation methodology, AI system robustness, and dependable LLM deployment.