Natural Language Questions as an Interface for Knowledge Graphs: QRAKEN Graph Distillation and Semantic Self-Healing

2026-10-06

AI Summary

This study addresses the limitation that SPARQL queries generated by large language models over unfamiliar knowledge graphs, while syntactically valid, frequently deviate from actual data distributions. To overcome this, we propose QRAKEN, a framework that constructs TTQL empirical patterns via graph distillation to guide LLM query generation and incorporates deterministic checks for semantic self-healing. This training-free, ontology-agnostic neuro-symbolic pipeline grounds queries using empirical graph evidence rather than relying solely on schema information. Evaluated on the CK25 benchmark, QRAKEN achieves a strict F1 score of 0.652, outperforming the strongest baseline by 32% and demonstrating substantial improvements over SHACL-based approaches.

Abstract

Natural-language access to RDF knowledge graphs is a core Semantic Web ambition. Large language models (LLMs) have advanced Text-to-SPARQL, yet on unfamiliar graphs they often generate valid queries that misrepresent the populated data model. QRAKEN is a training-free, ontology-agnostic neurosymbolic pipeline grounding generation in empirical graph evidence rather than schema expectations. An offline distiller produces TTQL, a compact description of populated multi-hop patterns, conditional frequencies and path-conditioned literal examples, plus a class-property co-occurrence matrix. Online, TTQL guides the LLM, while deterministic syntax, vocabulary and data-model checks provide diagnostics for iterative refinement. On CK25 (First International Text2SPARQL Challenge), under matched-condition recomputation on a QLever snapshot, QRAKEN achieves strict F1 of 0.643 $\pm$ 0.026 with GPT-4.1 mini and 0.652 $\pm$ 0.012 with GPT-5.4: relative gains of 30% and 32% over the strongest recomputed participant, outperforming systems using the same base model family. Ablations identify TTQL patterns as the dominant driver (+0.31 strict F1 over a shape-only baseline); the refinement loop provides a cheap safety net, rejecting triple patterns unsupported by the co-occurrence matrix. Compared with auto-derived SHACL, TTQL yields 64% higher strict F1, supporting the value of empirical patterns beyond schema exposure. With two local 35B 4-bit open-weight models at zero marginal cost, the same pipeline matches the strongest recomputed participant, and TTQL advantages over shape-only and SHACL baselines persist. Results on a single, relatively small benchmark provide an initial empirical signal; monolithic TTQL injection on very open cross-domain graphs remains the main limitation.

Problem

• Text-to-SPARQL
• Knowledge Graphs
• Large Language Models
• Natural Language Interface
• Data Model Misrepresentation

Innovation

• Text-to-SPARQL
• Graph Distillation
• Neurosymbolic Pipeline
• Semantic Self-Healing
• Training-free