Conformal Factuality Control for Multi-Hop Retrieval-Augmented Generation

2026-09-27

AI Summary

This study addresses the challenge that external evidence in multi-hop retrieval-augmented generation (RAG) often fails to guarantee the factual accuracy of generated claims. To tackle this, the work introduces split conformal prediction into multi-hop RAG pipelines built upon Llama 3.1 and GPT-4o-mini, implementing a conformal claim filtering mechanism. This research is the first to demonstrate that conformal factual control can be extended to multi-stage dependent reasoning scenarios, revealing that nominal reliability must be interpreted jointly with claim retention and abstention rates. Evaluated on datasets such as HotpotQA at a 95% target confidence level, the approach achieves fully supported response rates of 95.8%–97.2% while retaining only 4.4%–31.1% of claims, thereby substantially enhancing the controllability of generated facts.

Abstract

Retrieval-augmented generation (RAG) can ground large language models in external evidence, but retrieved context does not guarantee that generated claims are factually supported. This problem is especially relevant in multi-hop RAG, where retrieval and reasoning proceed through multiple dependent stages. We study whether claim-level conformal factuality control, previously developed for RAG, remains effective in this setting. We apply split-conformal claim filtering to multi-hop RAG and evaluate it on HotpotQA, Natural Questions, and TriviaQA using Llama 3.1 8B and GPT-4o-mini, together with a single-hop reference experiment. Across all six multi-hop model-dataset configurations, increasingly stringent conformal targets consistently increase the fraction of responses whose retained claims are fully supported. At the 95% target, this rate ranges from 95.80% to 97.20%, compared with 55.60%-76.03% without filtering. However, the improvement is strongly selective: only 4.41%-31.09% of generated claims are retained and 9.70%-51.40% of responses remain non-empty at the 95% target. These results show that conformal factuality extends to multi-hop RAG, while demonstrating that nominal reliability must be interpreted jointly with claim retention and abstention.

Problem

• Multi-hop RAG
• Conformal factuality control
• Retrieval-augmented generation
• Claim filtering
• Factual reliability

Innovation

• Multi-Hop RAG
• Conformal Factuality Control
• Split-Conformal Claim Filtering
• Retrieval-Augmented Generation
• Abstention