Technology Category
Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP ModelsPlanning, Routing, and Scheduling: Planning with Language ModelsMachine Learning: Large Multimodal Models (LMMs)
The Design of an LLM-powered Unstructured Analytics System
🤖 AI Summary
To address the low accuracy and poor interpretability of semantic analysis for unstructured documents, this paper proposes Aryn: a task-oriented LLM system architecture paradigm. Methodologically, Aryn introduces a novel deep coupling between a declarative document processing engine (Sycamore) and a natural-language-to-code query planner (Luna), integrated with the DocParse parser to form an end-to-end analytical pipeline. This design overcomes the accuracy limitations of conventional RAG systems in complex semantic reasoning, enabling natural-language querying, automatic generation of executable semantic plans, and full execution provenance with intermediate-result visualization. Evaluated on real-world NTSB aviation accident reports, Aryn achieves significantly higher accuracy than baseline RAG approaches while substantially improving user trust and debugging capability.
Technology Category
Application Category
📝 Abstract
LLMs demonstrate an uncanny ability to process unstructured data, and as such, have the potential to go beyond search and run complex, semantic analyses at scale. We describe the design of an unstructured analytics system, Aryn, and the tenets and use cases that motivate its design. With Aryn, users specify queries in natural language and the system automatically determines a semantic plan and executes it to compute an answer from a large collection of unstructured documents. At the core of Aryn is Sycamore, a declarative document processing engine, that provides a reliable distributed abstraction called DocSets. Sycamore allows users to analyze, enrich, and transform complex documents at scale. Aryn includes Luna, a query planner that translates natural language queries to Sycamore scripts, and DocParse, which takes raw PDFs and document images, and converts them to DocSets for downstream processing. We show how these pieces come together to achieve better accuracy than RAG on analytics queries over real world reports from the National Transportation Safety Board (NTSB). Also, given current limitations of LLMs, we argue that an analytics system must provide explainability to be practical, and show how Aryn's user interface does this to help build trust.
Problem
Research questions and friction points this paper is trying to address.
Non-structured Information Analysis
Data Extraction
Efficiency and Accuracy Improvement
Innovation
Methods, ideas, or system contributions that make the work stand out.
Aryn
large language models
unstructured information analysis
🔎 Similar Papers
No similar papers found.
