A Human-in-the-Loop, LLM-Centered Architecture for Knowledge-Graph Question Answering

2026-02-05

AI Summary

This work addresses the challenges of hallucination, outdated information, and lack of interpretability in large language models (LLMs) when applied to knowledge-intensive tasks, as well as the barrier posed by specialized query languages that limit non-expert access to knowledge graphs. The authors propose a novel human-in-the-loop framework that integrates LLMs with graph query generation: natural language questions are translated into interpretable Cypher queries, which users can iteratively refine through natural language feedback to achieve accurate and transparent knowledge graph access. By combining the usability of natural language with the semantic rigor of knowledge graphs, the approach demonstrates strong performance in multi-hop reasoning and error detection, validated on a movie-based benchmark comprising 90 queries as well as real-world knowledge graphs from Hyena and MaRDI.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large Language Models (LLMs) excel at language understanding but remain limited in knowledge-intensive domains due to hallucinations, outdated information, and limited explainability. Text-based retrieval-augmented generation (RAG) helps ground model outputs in external sources but struggles with multi-hop reasoning. Knowledge Graphs (KGs), in contrast, support precise, explainable querying, yet require a knowledge of query languages. This work introduces an interactive framework in which LLMs generate and explain Cypher graph queries and users iteratively refine them through natural language. Applied to real-world KGs, the framework improves accessibility to complex datasets while preserving factual accuracy and semantic rigor and provides insight into how model performance varies across domains. Our core quantitative evaluation is a 90-query benchmark on a synthetic movie KG that measures query explanation quality and fault detection across multiple LLMs, complemented by two smaller real-life query-generation experiments on a Hyena KG and the MaRDI (Mathematical Research Data Initiative) KG.

Problem

• Knowledge Graph Question Answering
• Large Language Models
• Human-in-the-Loop
• Query Explainability
• Hallucination

Innovation

• Human-in-the-Loop
• LLM-Centered Architecture
• Knowledge Graph Question Answering
• Cypher Query Generation
• Explainable AI