Advanced System Integration: Analyzing OpenAPI Chunking for Retrieval-Augmented Generation

2024-11-29

arXiv.org

AI Summary

To address challenges in dynamic system integration—including difficulty in API endpoint discovery and high token overhead in retrieval-augmented generation (RAG)—this paper proposes a summary-based Discovery Agent architecture. First, OpenAPI specifications are processed via format-aware, LLM-driven intelligent chunking to generate semantically compact endpoint summaries. Second, an efficient retrieval pipeline combines summary-based pre-filtering with on-demand loading of fine-grained details. This work presents the first systematic evaluation of OpenAPI chunking strategies for RAG-based endpoint discovery. On the RestBench benchmark, our method significantly improves F1 score (+12.3%) and precision (+15.6%) over naive chunking, maintains high recall (92.4%), and reduces token consumption by 68%. Key contributions include: (1) a format-aware chunking paradigm for OpenAPI documents; (2) an endpoint-level summary pre-filtering mechanism; and (3) a lightweight, efficient RAG framework tailored for API endpoint discovery.

Technology Category

• Natural Language Processing: Summarization
• Search and Optimization: Distributed Search
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Integrating multiple (sub-)systems is essential to create advanced Information Systems (ISs). Difficulties mainly arise when integrating dynamic environments across the IS lifecycle. A traditional approach is a registry that provides the API documentation of the systems' endpoints. Large Language Models (LLMs) have shown to be capable of automatically creating system integrations (e.g., as service composition) based on this documentation but require concise input due to input token limitations, especially regarding comprehensive API descriptions. Currently, it is unknown how best to preprocess these API descriptions. Within this work, we (i) analyze the usage of Retrieval Augmented Generation (RAG) for endpoint discovery and the chunking, i.e., preprocessing, of OpenAPIs to reduce the input token length while preserving the most relevant information. To further reduce the input token length for the composition prompt and improve endpoint retrieval, we propose (ii) a Discovery Agent that only receives a summary of the most relevant endpoints and retrieves details on demand. We evaluate RAG for endpoint discovery using the RestBench benchmark, first, for the different chunking possibilities and parameters measuring the endpoint retrieval recall, precision, and F1 score. Then, we assess the Discovery Agent using the same test set. With our prototype, we demonstrate how to successfully employ RAG for endpoint discovery to reduce the token count. While revealing high values for recall, precision, and F1, further research is necessary to retrieve all requisite endpoints. Our experiments show that for preprocessing, LLM-based and format-specific approaches outperform na""ive chunking methods. Relying on an agent further enhances these results as the agent splits the tasks into multiple fine granular subtasks, improving the overall RAG performance in the token count, precision, and F1 score.

Problem

• Optimizing OpenAPI chunking for Retrieval-Augmented Generation input efficiency
• Reducing token length in API descriptions while preserving key information
• Enhancing endpoint discovery precision using a dynamic Discovery Agent

Innovation

• Analyzing RAG for OpenAPI chunking optimization
• Proposing Discovery Agent for endpoint retrieval
• Evaluating chunking methods with RestBench benchmark