Extract Information from Hybrid Long Documents Leveraging LLMs: A Framework and Dataset

2024-12-28

AI Summary

Hybrid long documents (HLDs)—comprising extremely lengthy textual content and complex tabular structures—exceed the context window limitations of large language models (LLMs), hindering accurate structured information extraction. Method: This paper proposes the Automated Information Extraction (AIE) framework, featuring: (1) the first systematic evaluation of LLMs’ comprehension capabilities on HLDs; (2) a lightweight table serialization technique preserving semantic fidelity; (3) an adaptive chunking and summarization strategy to mitigate context bottlenecks; and (4) domain-specific prompt templates tailored for financial statements. Contributions/Results: We introduce FINE, the first benchmark dataset for numerical extraction from HLDs; publicly release the AIE framework and source code; and demonstrate substantial improvements in numerical and structured information extraction accuracy across diverse scenarios, empirically validating AIE’s robust adaptability to heterogeneous long documents.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models (LLMs) demonstrate exceptional performance in textual understanding and tabular reasoning tasks. However, their ability to comprehend and analyze hybrid text, containing textual and tabular data, remains unexplored. The hybrid text often appears in the form of hybrid long documents (HLDs), which far exceed the token limit of LLMs. Consequently, we apply an Automated Information Extraction framework (AIE) to enable LLMs to process the HLDs and carry out experiments to analyse four important aspects of information extraction from HLDs. Given the findings: 1) The effective way to select and summarize the useful part of a HLD. 2) An easy table serialization way is enough for LLMs to understand tables. 3) The naive AIE has adaptability in many complex scenarios. 4) The useful prompt engineering to enhance LLMs on HLDs. To address the issue of dataset scarcity in HLDs and support future work, we also propose the Financial Reports Numerical Extraction (FINE) dataset. The dataset and code are publicly available in the attachments.

Problem

• Large Language Models
• Mixed Long Documents
• Complex Structure Handling

Innovation

• Automatic Information Extraction (AIE)
• Large Language Models (LLMs)
• Financial Reports Numerical Extraction (FINE) Dataset