PaperTrail: A Claim-Evidence Interface for Grounding Provenance in LLM-based Scholarly Q&A

2026-02-24

AI Summary

Large language models frequently generate unsupported assertions or omit critical information in academic question answering, while existing provenance mechanisms offer only coarse-grained traceability insufficient for scholarly rigor. This work proposes PaperTrail, a system that introduces, for the first time in academic QA, a fine-grained claim–evidence alignment mechanism: it employs natural language processing to extract claims from model responses and evidence units from source literature, then establishes explicit mappings between them. An interactive visualization interface enables users to inspect the support status—supported, unsupported, or missing—for each claim. User studies reveal that PaperTrail significantly reduces users’ trust in model outputs yet fails to substantially diminish their reliance on them, highlighting the critical role of cognitive load in human–AI collaboration.

Technology Category

• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Natural Language Processing: Question Answering
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Large language models (LLMs) are increasingly used in scholarly question-answering (QA) systems to help researchers synthesize vast amounts of literature. However, these systems often produce subtle errors (e.g., unsupported claims, errors of omission), and current provenance mechanisms like source citations are not granular enough for the rigorous verification that scholarly domain requires. To address this, we introduce PaperTrail, a novel interface that decomposes both LLM answers and source documents into discrete claims and evidence, mapping them to reveal supported assertions, unsupported claims, and information omitted from the source texts. We evaluated PaperTrail in a within-subjects study with 26 researchers who performed two scholarly editing tasks using PaperTrail and a baseline interface. Our results show that PaperTrail significantly lowered participants'trust compared to the baseline. However, this increased caution did not translate to behavioral changes, as people continued to rely on LLM-generated scholarly edits to avoid a cognitively burdensome task. We discuss the value of claim-evidence matching for understanding LLM trustworthiness in scholarly settings, and present design implications for cognition-friendly communication of provenance information.

Problem

• large language models
• scholarly question-answering
• provenance
• claim-evidence
• trustworthiness

Innovation

• claim-evidence alignment
• provenance visualization
• scholarly QA
• LLM trustworthiness
• fine-grained attribution