🤖 AI Summary
This study addresses the complexity of evaluating Retrieval-Augmented Generation (RAG) systems—stemming from their multi-component architecture (indexing, retrieval, generation) and high-dimensional parameter space, which hinder quantitative quality assessment. Through a systematic review of 63 papers, we construct a full-stack evaluation taxonomy spanning four dimensions: datasets, retrievers, indexes/databases, and generators. We pioneer the dual role of large language models (LLMs) in RAG evaluation: (i) automating the construction of high-quality, diverse test suites, and (ii) performing interpretable, multi-faceted assessments—including retrieval relevance, answer faithfulness, and information completeness. We rigorously delineate the boundaries and synergies between LLM-based automated evaluation and human judgment. Empirical validation across multiple benchmarks confirms the framework’s effectiveness, and we deliver a reusable, production-ready RAG evaluation guideline for industry practitioners.
📝 Abstract
Retrieval-Augmented Generation (RAG) has advanced significantly in recent years. The complexity of RAG systems, which involve multiple components-such as indexing, retrieval, and generation-along with numerous other parameters, poses substantial challenges for systematic evaluation and quality enhancement. Previous research highlights that evaluating RAG systems is essential for documenting advancements, comparing configurations, and identifying effective approaches for domain-specific applications. This study systematically reviews 63 academic articles to provide a comprehensive overview of state-of-the-art RAG evaluation methodologies, focusing on four key areas: datasets, retrievers, indexing and databases, and the generator component. We observe the feasibility of an automated evaluation approach for each component of a RAG system, leveraging an LLM capable of both generating evaluation datasets and conducting evaluations. In addition, we found that further practical research is essential to provide companies with clear guidance on the do's and don'ts of implementing and evaluating RAG systems. By synthesizing evaluation approaches for key RAG components and emphasizing the creation and adaptation of domain-specific datasets for benchmarking, we contribute to the advancement of systematic evaluation methods and the improvement of evaluation rigor for RAG systems. Furthermore, by examining the interplay between automated approaches leveraging LLMs and human judgment, we contribute to the ongoing discourse on balancing automation and human input, clarifying their respective contributions, limitations, and challenges in achieving robust and reliable evaluations.