Towards a rigorous evaluation of RAG systems: the challenge of due diligence

2025-07-29

AI Summary

Retrieval-Augmented Generation (RAG) systems exhibit insufficient reliability in high-stakes domains—such as investment due diligence—where hallucinations, off-topic responses, and citation failures pose critical risks. Method: We propose a statistically rigorous yet industrially scalable evaluation framework that integrates human expert annotations with calibrated LLM-based judges in a collaborative assessment protocol. Inspired by predictive inference, our approach enables precise, quantitative identification of multiple failure modes—including hallucination, irrelevance, and retrieval failure. Contributions/Results: (1) We introduce the first RAG reliability benchmark specifically designed for fund due diligence; (2) we publicly release a high-quality evaluation dataset and open-source evaluation code; (3) empirical validation demonstrates that our framework significantly improves assessment reliability while enabling large-scale automated evaluation—establishing a reproducible, verifiable evaluation paradigm for deploying RAG systems in mission-critical applications.

Technology Category

Reasoning under Uncertainty: Other Foundations of Reasoning under UncertaintyMachine Learning: Calibration & Uncertainty QuantificationData Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGEconomics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAIWeb Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

The rise of generative AI, has driven significant advancements in high-risk sectors like healthcare and finance. The Retrieval-Augmented Generation (RAG) architecture, combining language models (LLMs) with search engines, is particularly notable for its ability to generate responses from document corpora. Despite its potential, the reliability of RAG systems in critical contexts remains a concern, with issues such as hallucinations persisting. This study evaluates a RAG system used in due diligence for an investment fund. We propose a robust evaluation protocol combining human annotations and LLM-Judge annotations to identify system failures, like hallucinations, off-topic, failed citations, and abstentions. Inspired by the Prediction Powered Inference (PPI) method, we achieve precise performance measurements with statistical guarantees. We provide a comprehensive dataset for further analysis. Our contributions aim to enhance the reliability and scalability of RAG systems evaluation protocols in industrial applications.

Problem

• Evaluating reliability of RAG systems in high-risk sectors
• Identifying system failures like hallucinations and off-topic responses
• Developing robust evaluation protocols with statistical guarantees

Innovation

• Combines human and LLM-Judge annotations
• Uses Prediction Powered Inference method
• Provides comprehensive dataset for analysis