Document Attribution: Examining Citation Relationships using Large Language Models

2025-05-09

AI Summary

This paper addresses the challenge of trustworthy attribution in large language models (LLMs) for document understanding—specifically, how to precisely trace model outputs back to source documents and assess the reliability of such citations. We propose a zero-shot textual entailment–based attribution method and conduct the first systematic analysis of how attention weights across Transformer layers influence attribution performance. Evaluated on the AttributionBench benchmark (both in-distribution and out-of-distribution splits) using Flan-UL2 and Flan-T5-small, our approach improves accuracy by 0.27% and 2.4%, respectively; notably, Flan-T5-small achieves significantly higher F1 scores than baselines across most layers. Our key contributions are: (1) a lightweight, fine-tuning–free attribution framework; (2) empirical characterization of layer-wise distribution of deep attribution capability within the attention mechanism; and (3) a reproducible, quantitatively evaluable methodology for interpretable document question answering.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

As Large Language Models (LLMs) are increasingly applied to document-based tasks - such as document summarization, question answering, and information extraction - where user requirements focus on retrieving information from provided documents rather than relying on the model's parametric knowledge, ensuring the trustworthiness and interpretability of these systems has become a critical concern. A central approach to addressing this challenge is attribution, which involves tracing the generated outputs back to their source documents. However, since LLMs can produce inaccurate or imprecise responses, it is crucial to assess the reliability of these citations. To tackle this, our work proposes two techniques. (1) A zero-shot approach that frames attribution as a straightforward textual entailment task. Our method using flan-ul2 demonstrates an improvement of 0.27% and 2.4% over the best baseline of ID and OOD sets of AttributionBench, respectively. (2) We also explore the role of the attention mechanism in enhancing the attribution process. Using a smaller LLM, flan-t5-small, the F1 scores outperform the baseline across almost all layers except layer 4 and layers 8 through 11.

Problem

• Ensuring trustworthiness of LLM-generated document attributions
• Evaluating reliability of citations in LLM outputs
• Improving attribution accuracy using zero-shot and attention techniques

Innovation

• Zero-shot textual entailment for document attribution
• Attention mechanism enhances citation reliability
• Improved F1 scores with flan-t5-small model