DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing

2024-10-16

arXiv.org

AI Summary

Large language models (LLMs) exhibit insufficient accuracy in analyzing complex, unstructured documents—particularly in identifying specific clauses (e.g., force majeure, indemnification) within lengthy legal texts—due to brittle multi-step reasoning and lack of systematic optimization. Method: We propose the first declarative document processing framework, integrating agent-driven query rewriting, task-adaptive logical rewriting instructions, synthetic verification prompt generation, and a Monte Carlo heuristic search algorithm to dynamically balance latency and accuracy. Unlike cost-centric approaches, our method prioritizes precision by automating end-to-end optimization of LLM reasoning chains. Contribution/Results: Evaluated on four legal and commercial document understanding tasks, our framework achieves 25–80% absolute accuracy gains over fine-tuned baselines. The implementation is open-sourced and has garnered over 1,700 GitHub stars.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Analyzing unstructured data has been a persistent challenge in data processing. Large Language Models (LLMs) have shown promise in this regard, leading to recent proposals for declarative frameworks for LLM-powered processing of unstructured data. However, these frameworks focus on reducing cost when executing user-specified operations using LLMs, rather than improving accuracy, executing most operations as-is (in a single LLM call). This is problematic for complex tasks and data, where LLM outputs for user-defined operations are often inaccurate, even with optimized prompts. For example, an LLM may struggle to identify {em all} instances of specific clauses, like force majeure or indemnification, in lengthy legal documents, requiring decomposition of the data, the task, or both. We present DocETL, a system that optimizes complex document processing pipelines, while accounting for LLM shortcomings. DocETL offers a declarative interface for users to define such pipelines and uses an agent-based approach to automatically optimize them, leveraging novel agent-based rewrites (that we call rewrite directives), as well as an optimization and evaluation framework. We introduce (i) logical rewriting of pipelines, tailored for LLM-based tasks, (ii) an agent-guided plan evaluation mechanism that synthesizes and orchestrates task-specific validation prompts, and (iii) an optimization algorithm that efficiently finds promising plans, considering the latencies of agent-based plan generation and evaluation. Our evaluation on four different unstructured document analysis tasks demonstrates that DocETL finds plans with outputs that are 25 to 80% more accurate than well-engineered baselines, addressing a critical gap in unstructured data analysis. DocETL is open-source at docetl.org, and as of March 2025, has amassed over 1.7k GitHub Stars, with users spanning a variety of domains.

Problem

• Improving accuracy in complex document processing with LLMs
• Automating optimization of document processing pipelines
• Addressing LLM shortcomings in unstructured data analysis

Innovation

• Agent-based query rewriting for document processing
• Logical pipeline optimization for LLM tasks
• Agent-guided plan evaluation with validation prompts