Evaluating Knowledge Graph Based Retrieval Augmented Generation Methods under Knowledge Incompleteness

2025-04-07

AI Summary

Knowledge graph (KG) incompleteness severely limits the performance of KG-augmented retrieval-augmented generation (KG-RAG). Method: This work presents the first systematic robustness evaluation of mainstream KG-RAG methods under incomplete KGs. We introduce a standardized evaluation framework for KG incompleteness, employing multiple controlled triple-deletion strategies to simulate realistic missing knowledge. Contribution/Results: Experiments reveal that even modest incompleteness—5%–20% of triples removed—causes substantial degradation in question-answering accuracy, with an average drop of 23.6%, confirming KG completeness as a critical bottleneck in current KG-RAG systems. Our study bridges a key gap in existing benchmarks, which largely ignore KG incompleteness, and provides empirically grounded insights and a reproducible evaluation paradigm to guide the development of robust KG-RAG methods.

Technology Category

Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB CompletionKnowledge Representation and Reasoning: Computational Complexity of ReasoningNatural Language Processing: Question Answering

Application Category

Search and Retrieval-Augmented AI: Web evaluation methodologies and metricsSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsGraph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Knowledge Graph based Retrieval-Augmented Generation (KG-RAG) is a technique that enhances Large Language Model (LLM) inference in tasks like Question Answering (QA) by retrieving relevant information from knowledge graphs (KGs). However, real-world KGs are often incomplete, meaning that essential information for answering questions may be missing. Existing benchmarks do not adequately capture the impact of KG incompleteness on KG-RAG performance. In this paper, we systematically evaluate KG-RAG methods under incomplete KGs by removing triples using different methods and analyzing the resulting effects. We demonstrate that KG-RAG methods are sensitive to KG incompleteness, highlighting the need for more robust approaches in realistic settings.

Problem

• Evaluating KG-RAG performance under incomplete knowledge graphs
• Assessing impact of missing KG triples on retrieval-augmented generation
• Identifying sensitivity of KG-RAG methods to knowledge incompleteness

Innovation

• KG-RAG enhances LLM with knowledge graphs
• Evaluates KG-RAG under incomplete knowledge graphs
• Proposes robust methods for incomplete KG scenarios