🤖 AI Summary
This work addresses the problem of cross-source confusion in large language model agents under the Model-Context-Prompt (MCP) architecture, where factual errors in responses are incorrectly attributed to irrelevant evidence sources. The paper introduces the first source-aware factuality verification framework, which parses MCP execution traces to decompose responses into atomic claims, routes each claim to its corresponding evidence source, and evaluates claim support through natural language inference and token alignment. Crucially, it compares the claimed source against the actual evidence source, enabling per-claim and holistic allow/block decisions. By treating source attribution as an independent dimension of factuality, the framework supports detection of misattribution and enables an automated retrieval-augmented repair loop. Evaluated on 281 medical MCP trajectories, it achieves a held-out blocking F1 of 0.802 and source accuracy of 0.858; in multi-source settings, blocking F1 improves to 0.846, with 100% detection and correction of injected attribution manipulations.
📝 Abstract
Tool-using LLM agents increasingly use the Model Context Protocol (MCP) to answer from heterogeneous evidence sources, including search, APIs, databases, clinical records, and formulary tools. Standard factuality metrics usually test whether an answer is supported by pooled evidence, missing a provenance-sensitive failure mode: a claim may be supported somewhere while being attributed to the wrong source. We call this cross-source conflation.
We introduce ProvenanceGuard, a source-aware verifier for MCP-grounded answers. It consumes captured MCP traces with stable tool IDs, source IDs, and raw outputs; decomposes answers into atomic claims; routes claims to source-specific evidence; checks support with NLI and a token-alignment proxy; compares stated attribution with the routed source; and returns per-claim verdicts plus an answer-level allow/block decision. Blocked answers can be repaired with retrieval-augmented answer revision and re-verified.
We evaluate on 281 medical-domain MCP-agent traces. A 266-trace adjudicated subset yields 2,325 LLM-assisted claim labels split by trace; 361 held-out labels are human-verified. On the 40-trace held-out split, ProvenanceGuard achieves block F1 0.802 and source accuracy 0.858 over 260 source-eligible claims, outperforming source-blind baselines that do not emit claim-to-source IDs. On a harder multi-source benchmark it reaches block F1 0.846, while source-plus-relation accuracy drops to 0.229, showing that exact source ownership remains difficult with semantically close sources. Repair-and-reverify resolves all blocked answers in the full trace set, often via conservative fallback. In 50 controlled clinical conflation probes, ProvenanceGuard detects all injected attribution swaps with no retained wrong attribution. These results show that source attribution is an independent axis for factuality verification in MCP-based agents.