RAGPulse: An Open-Source RAG Workload Trace to Optimize RAG Serving Systems

2025-11-17

AI Summary

Existing LLM inference trace datasets fail to capture realistic workload characteristics of RAG systems—particularly their multi-stage (retrieval-generation) execution flow and knowledge dependency—leading to a significant performance gap between research and industrial deployment. This paper introduces RAGPulse: the first high-fidelity RAG workload trace dataset derived from a production campus QA system serving over 40,000 faculty and students. Our approach systematically collects and anonymizes real-world query-execution traces, preserving critical temporal and access patterns. Key contributions include: (1) the first empirical demonstration of strong temporal locality and highly skewed document access distributions in RAG workloads; (2) a hash-based privacy-preserving trace format enabling content-aware batching and retrieval caching modeling; and (3) open-sourcing both the dataset and analysis toolkit to establish an empirical foundation for RAG performance modeling, optimization, and benchmarking.

Technology Category

• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Retrieval-Augmented Generation (RAG) is a critical paradigm for building reliable, knowledge-intensive Large Language Model (LLM) applications. However, the multi-stage pipeline (retrieve, generate) and unique workload characteristics (e.g., knowledge dependency) of RAG systems pose significant challenges for serving performance optimization. Existing generic LLM inference traces fail to capture these RAG-specific dynamics, creating a significant performance gap between academic research and real-world deployment. To bridge this gap, this paper introduces RAGPulse, an open-source RAG workload trace dataset. This dataset was collected from an university-wide Q&A system serving that has served more than 40,000 students and faculties since April 2024. We detail RAGPulse's system architecture, its privacy-preserving hash-based data format, and provide an in-depth statistical analysis. Our analysis reveals that real-world RAG workloads exhibit significant temporal locality and a highly skewed hot document access pattern. RAGPulse provides a high-fidelity foundation for researchers to develop and validate novel optimization strategies for RAG systems, such as content-aware batching and retrieval caching, ultimately enhancing the efficiency and reliability of RAG services. The code is available at https://github.com/flashserve/RAGPulse.

Problem

• Existing LLM traces fail to capture RAG-specific workload dynamics
• RAG systems face performance challenges from multi-stage pipelines
• There is a performance gap between research and real-world RAG deployment

Innovation

• Open-source RAG workload trace dataset
• Collected from university Q&A system
• Enables optimization via content-aware batching and caching