🤖 AI Summary
Current RAG system evaluations overly rely on end-to-end accuracy, failing to capture enterprise-level requirements across dimensions such as reasoning complexity, retrieval difficulty, document structural diversity, and interpretability. Consequently, models achieving high scores often exhibit insufficient reliability in real-world deployments. To address this gap, this work proposes the first difficulty taxonomy integrating these four dimensions and introduces a multidimensional diagnostic framework and benchmark tailored for enterprise applications. The framework systematically identifies weaknesses of RAG systems in complex, realistic settings and effectively exposes performance bottlenecks that hinder practical deployment, thereby offering actionable pathways for evaluation and optimization to enhance real-world reliability.
📝 Abstract
Performance evaluation of Retrieval-Augmented Generation (RAG) systems within enterprise environments is governed by multi-dimensional and composite factors extending far beyond simple final accuracy checks. These factors include reasoning complexity, retrieval difficulty, the diverse structure of documents, and stringent requirements for operational explainability. Existing academic benchmarks fail to systematically diagnose these interlocking challenges, resulting in a critical gap where models achieving high performance scores fail to meet the expected reliability in practical deployment.
To bridge this discrepancy, this research proposes a multi-dimensional diagnostic framework by defining a four-axis difficulty taxonomy and integrating it into an enterprise RAG benchmark to diagnose potential system weaknesses.