Context-Aware Hierarchical Merging for Long Document Summarization

2025-02-03

AI Summary

To address hallucination accumulation and factual inconsistency arising from hierarchical merging in abstractive summarization of ultra-long documents (>100K tokens), this paper proposes a context-aware hierarchical merging paradigm. The method systematically introduces three novel context-enhancement mechanisms: (i) key-segment replacement, (ii) prompt-guided summary refinement, and (iii) implicit provenance via citation alignment. Empirical analysis identifies the combination of refinement and extractive key-segment selection as optimal. Evaluated on legal and narrative domain datasets using Llama 3.1-based models, the approach significantly outperforms zero-shot and standard hierarchical merging baselines. It achieves simultaneous improvements in factual consistency and coherence, empirically validating the critical role of source-document context at every summarization layer.

Technology Category

• Natural Language Processing: Summarization
• Data Mining & Knowledge Management: Data Visualization & Summarization
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

Hierarchical Merging is a technique commonly used to summarize very long texts ($>$100K tokens) by breaking down the input into smaller sections, summarizing those sections individually, and then merging or combining those summaries into a final coherent summary. Although it helps address the limitations of large language models (LLMs) with fixed input length constraints, the recursive merging process can amplify LLM hallucinations, increasing the risk of factual inaccuracies. In this paper, we seek to mitigate hallucinations by enriching hierarchical merging with context from the source document. Specifically, we propose different approaches to contextual augmentation ranging from emph{replacing} intermediate summaries with relevant input context, to emph{refining} them while using the context as supporting evidence, and emph{aligning} them implicitly (via citations) to the input. Experimental results on datasets representing legal and narrative domains show that contextual augmentation consistently outperforms zero-shot and hierarchical merging baselines for the Llama 3.1 model family. Our analysis further reveals that refinement methods tend to perform best when paired with extractive summarization for identifying relevant input.

Problem

• Hierarchical Aggregation
• Inaccuracy
• Error Amplification

Innovation

• Hierarchical Aggregation
• Content-driven Error Reduction
• Key Information Utilization