🤖 AI Summary
This study addresses the lack of empirical validation for existing retrieval-augmented generation (RAG) evaluation metrics in real-world scenarios. Leveraging a human-annotated commercial question-answering dataset, it presents the first systematic comparison of prominent metrics from four major evaluation frameworks—Ragas, DeepEval, RAGChecker, and Opik. Through correlation analyses against human judgments and traditional metrics such as recall, the work reveals that most automatic metrics exhibit weak alignment with human assessments. These findings not only highlight significant limitations in current RAG evaluation methodologies when applied to practical settings but also provide empirical grounding and actionable directions for developing more reliable, real-world-oriented evaluation approaches.
📝 Abstract
This paper reports an empirical study evaluating the relevance of several RAG metrics. The experiment is based on a question-answering dataset created by human annotators from business data. The generated responses and retrieved spans of a RAG system are scored using evaluation metrics from four libraries (Ragas, DeepEval, RAGChecker, Opik). These metrics are compared to scores given by two evaluators, as well as to standard metrics such as recall. An analysis of correlations is conducted. Finally, we highlight certain limitations of our methodology, compare it to those used in the literature, and suggest some avenues for future research. This paper is an English translation of a paper originally published in the French-speaking workshop EvalLLM (Brabant, 2026).