LLM-based SPARQL Query Generation from Natural Language over Federated Knowledge Graphs

2024-10-08

arXiv.org

AI Summary

Natural language-to-SPARQL translation in bioinformatics federated knowledge graphs suffers from high error rates and severe large language model (LLM) hallucination. Method: This paper proposes a retrieval-augmented generation (RAG) framework that integrates KG metadata—including schema structure and high-quality SPARQL query examples—via three tightly coupled components: schema-aware retrieval, example-driven generation, and executability verification (encompassing syntactic validation, semantic consistency checking, and automated correction). Contribution/Results: Experiments demonstrate substantial improvements in query accuracy and robustness, achieving state-of-the-art (SOTA) performance on real-world federated KG benchmarks. The framework has been deployed on the Expasy platform (chat.expasy.org), enabling reliable biomedical knowledge question answering and exploratory querying in production settings.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Description Logics
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We introduce a Retrieval-Augmented Generation (RAG) system for translating user questions into accurate federated SPARQL queries over bioinformatics knowledge graphs (KGs) leveraging Large Language Models (LLMs). To enhance accuracy and reduce hallucinations in query generation, our system utilises metadata from the KGs, including query examples and schema information, and incorporates a validation step to correct generated queries. The system is available online at chat.expasy.org.

Problem

• Generate SPARQL queries from natural language
• Improve accuracy in federated knowledge graphs
• Reduce hallucinations using metadata validation

Innovation

• LLM-based SPARQL query generation
• Retrieval-Augmented Generation system
• Metadata-enhanced query validation