🤖 AI Summary
Retrieval-augmented generation (RAG) systems often produce hallucinations, yet principled geometric diagnostics for semantic fidelity remain lacking. Method: We propose the Semantic Geometry Index (SGI), defined as the ratio of angular distances between the question-answer embedding pair and the question-context pair on the unit hypersphere—establishing the first theoretically grounded, spherical-geometric criterion for RAG semantic faithfulness. We identify and formalize “semantic inertia”: hallucinated responses exhibit smaller angular distances to the question than to the retrieved context. Contribution/Results: We prove that SGI’s discriminative power increases with question-context angular separation. Empirically, SGI achieves strong correlation (r = 0.85) with hallucination scores across models in HaluEval, high AUC (0.83) in high-separation regimes, and low expected calibration error (ECE = 0.10). Crucially, SGI robustly distinguishes semantic dependency (AUC = 0.83) from factual correctness (TruthfulQA AUC = 0.478), demonstrating both theoretical rigor and cross-dataset robustness.
📝 Abstract
When retrieval-augmented generation (RAG) systems hallucinate, what geometric trace does this leave in embedding space? We introduce the Semantic Grounding Index (SGI), defined as the ratio of angular distances from the response to the question versus the context on the unit hypersphere $mathbb{S}^{d-1}$.Our central finding is emph{semantic laziness}: hallucinated responses remain angularly proximate to questions rather than departing toward retrieved contexts. On HaluEval ($n$=5,000), we observe large effect sizes (Cohen's $d$ ranging from 0.92 to 1.28) across five embedding models with mean cross-model correlation $r$=0.85. Crucially, we derive from the spherical triangle inequality that SGI's discriminative power should increase with question-context angular separation $θ(q,c)$-a theoretical prediction confirmed empirically: effect size rises monotonically from $d$=0.61 -low $θ(q,c)$, to $d$=1.27 -high $θ(q,c)$, with AUC improving from 0.72 to 0.83. Subgroup analysis reveals that SGI excels on long responses ($d$=2.05) and short questions ($d$=1.22), while remaining robust across context lengths. Calibration analysis yields ECE=0.10, indicating SGI scores can serve as probability estimates, not merely rankings. A critical negative result on TruthfulQA (AUC=0.478) establishes that angular geometry measures topical engagement rather than factual accuracy. SGI provides computationally efficient, theoretically grounded infrastructure for identifying responses that warrant verification in production RAG deployments.
