SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation

2026-05-05

AI Summary

This work addresses the challenge in retrieval-augmented generation (RAG) where retrieved passages are topically relevant yet lack sufficient evidential support for accurate answer generation. To tackle this, the authors propose SURE-RAG, a novel framework that models evidence sufficiency as a set-level property for the first time. The approach employs a shared pairwise claim-evidence verifier to produce local relational distributions, which are then aggregated into interpretable global signals—such as coverage, conflict, and uncertainty—and integrated with a calibration mechanism and a three-way decision strategy to generate answers only when evidence is deemed sufficient. Evaluated on HotpotQA-RAG v3, SURE-RAG achieves a Macro-F1 score of 0.9075, significantly outperforming baseline methods, reducing risk by 37%, and demonstrating full auditability while surpassing GPT-4o in controlled verification tasks.

Abstract

Retrieval-augmented generation (RAG) grounds answers in retrieved passages, but retrieval is not verification: a passage can be topical and still fail to justify the answer. We frame this gap as evidence sufficiency verification for selective RAG answering: given a question, a candidate answer, and retrieved evidence, predict whether the evidence supports, refutes, or is insufficient, and abstain unless support is established. We present SURE-RAG, a transparent aggregation protocol built on the observation that evidence sufficiency is a set-level property: missing hops and unresolved conflicts cannot be detected by independent passage scoring. A shared pair-level claim-evidence verifier produces local relation distributions, which SURE-RAG aggregates into interpretable answer-level signals -- coverage, relation strength, disagreement, conflict, and retrieval uncertainty -- yielding a three-way decision and an auditable selective score. We evaluate on HotpotQA-RAG v3, a controlled multi-hop benchmark, under an artifact-aware protocol (shortcut baselines, counterfactual swaps, no-oracle checks, GPT-4o audits). Calibrated SURE-RAG reaches 0.9075 Macro-F1 (0.8951 +/- 0.0069), substantially above DeBERTa mean-pooling (0.6516) and a GPT-4o judge (0.7284), while matching a strong but opaque concat cross-encoder (0.8888 +/- 0.0109) with full auditability. Risk at 30% coverage drops from 0.2588 to 0.1642, a 37% reduction in unsafe answers. To deliberately probe the task boundary, we further contrast SURE-RAG with GPT-4o on HaluBench unsafe detection: the ranking reverses (0.3343 vs 0.7389 unsafe-F1), establishing that controlled sufficiency verification and natural hallucination detection are distinct problems.

Problem

• Retrieval-Augmented Generation
• Evidence Sufficiency
• Selective Answering
• Uncertainty-Aware Verification
• Hallucination Detection

Innovation

• evidence sufficiency
• selective RAG
• uncertainty-aware verification
• interpretable aggregation
• retrieval-augmented generation