ProvenAI: Provenance-Native Traces of Evidence in Generated Answers

2026-06-24

AI Summary

This work addresses the lack of verifiable mechanisms in existing retrieval-augmented systems for assessing the true influence of cited sources on generated answers. The authors propose a seven-stage framework that establishes three measurable layers of transparency in multi-hop question answering: answer correctness, citation fidelity, and single-document influence. For the first time, they integrate causal mediation analysis with database provenance theory to formally characterize the “citation-influence gap” and define faithfulness conditions via token-level KL divergence. Leveraging techniques such as citation-aware generation, attribution auditing, and leave-one-out interventions, the system achieves 53.53% answer accuracy and 71.55% average citation fidelity on the HotpotQA validation set, revealing a frequent misalignment between cited references and their actual evidential impact.

Abstract

Retrieval-augmented systems routinely present citations alongside generated answers, yet a citation does not confirm that the corresponding source meaningfully shaped the output. This paper introduces ProvenAI, a framework that decomposes transparency in multi-hop question answering into three independently measurable layers: answer correctness, citation fidelity against benchmark supporting evidence, and per-document influence under leave-one-resource-out intervention. Targeting the HotpotQA distractor benchmark through a seven-stage pipeline covering data normalisation, retrieval indexing, citation-aware answer generation, attribution auditing, ablation-based influence estimation, batch evaluation, and interactive inspection, ProvenAI evaluates 7,405 validation examples drawn from a canonical corpus of 509,300 passages. The system achieves 53.53% answer accuracy alongside a mean citation-fidelity score of 71.55%, and a worked example surfaces what we call the citation-influence gap: a clean citation audit co-occurring with a profile in which one cited source registers only weak influence while seven uncited sources demonstrably shift the output. We formalise the relationship between the implemented surface proxy and a token-level KL-divergence target through a stated faithfulness condition, ground the framework in causal-mediation analysis and database-provenance theory, and discuss how the three measurement layers compose with cryptographic provenance architectures emerging in autonomous scientific discovery. ProvenAI establishes that meaningful transparency in retrieval-grounded QA requires traceable links across retrieved, cited, and behaviourally influential evidence as three distinct, independently measured layers.

Problem

• provenance
• retrieval-augmented generation
• citation fidelity
• influence attribution
• transparency

Innovation

• provenance
• citation fidelity
• influence estimation
• retrieval-augmented generation
• causal mediation