SPAR: Session-based Pipeline for Adaptive Retrieval on Legacy File Systems

2025-12-14

AI Summary

Legacy enterprise document systems suffer from unstructured historical data and absent semantic indexing, resulting in inefficient, error-prone retrieval and poor decision-support capability. To address this, we propose a lightweight, two-stage adaptive retrieval framework: (1) constructing a semantic metadata index, followed by (2) session-aware dynamic vector library generation—avoiding costly full-document vectorization. Our novel “session-driven dynamic vector library generation” mechanism enhances retrieval accuracy while improving system controllability and deployment feasibility. We theoretically prove its computational complexity is significantly lower than conventional RAG, thereby increasing transparency. The method integrates large language models (LLMs), semantic modeling, and session-aware retrieval. Evaluated on a synthetic biomedical literature dataset emulating enterprise-scale data, our approach achieves substantial gains in retrieval accuracy and downstream task performance, demonstrating both efficiency and practical applicability.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Distributed Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

The ability to extract value from historical data is essential for enterprise decision-making. However, much of this information remains inaccessible within large legacy file systems that lack structured organization and semantic indexing, making retrieval and analysis inefficient and error-prone. We introduce SPAR (Session-based Pipeline for Adaptive Retrieval), a conceptual framework that integrates Large Language Models (LLMs) into a Retrieval-Augmented Generation (RAG) architecture specifically designed for legacy enterprise environments. Unlike conventional RAG pipelines, which require costly construction and maintenance of full-scale vector databases that mirror the entire file system, SPAR employs a lightweight two-stage process: a semantic Metadata Index is first created, after which session-specific vector databases are dynamically generated on demand. This design reduces computational overhead while improving transparency, controllability, and relevance in retrieval. We provide a theoretical complexity analysis comparing SPAR with standard LLM-based RAG pipelines, demonstrating its computational advantages. To validate the framework, we apply SPAR to a synthesized enterprise-scale file system containing a large corpus of biomedical literature, showing improvements in both retrieval effectiveness and downstream model accuracy. Finally, we discuss design trade-offs and outline open challenges for deploying SPAR across diverse enterprise settings.

Problem

• Efficiently retrieving unstructured data from legacy file systems
• Reducing computational costs of retrieval-augmented generation in enterprises
• Improving accuracy and relevance in historical data analysis

Innovation

• Integrates LLMs into RAG for legacy file systems
• Uses lightweight two-stage process with dynamic vector databases
• Improves retrieval transparency, controllability, and relevance