Towards Requirements Engineering for RAG Systems

2025-05-12

AI Summary

This paper addresses the requirements engineering bottleneck hindering the deployment of Retrieval-Augmented Generation (RAG) systems in high-reliability vertical domains—specifically maritime services. It introduces the first empirically grounded RAG-specific requirements engineering process model. Based on a case study with a maritime service provider, the research uncovers a fundamental tension between users’ expectations of “AI perfection” and the actual correctness of generated outputs, and reveals that “retrieval requirements” are highly context-sensitive—necessitating joint definition of “retrieval correctness” by domain experts and iterative experimental validation. The method integrates longitudinal case study, human-in-the-loop verification, and domain-adapted RAG evaluation to yield a reusable requirements elicitation process, a structured framework for specifying retrieval requirements, and a strategy for managing system limitations. This work transcends conventional LLM-centric requirements analysis paradigms, providing a rigorous methodological foundation for robust RAG deployment in safety-critical domains.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Metareasoning and Metaheuristics
• Knowledge Representation and Reasoning: Knowledge Engineering

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This short paper explores how a maritime company develops and integrates large-language models (LLM). Specifically by looking at the requirements engineering for Retrieval Augmented Generation (RAG) systems in expert settings. Through a case study at a maritime service provider, we demonstrate how data scientists face a fundamental tension between user expectations of AI perfection and the correctness of the generated outputs. Our findings reveal that data scientists must identify context-specific"retrieval requirements"through iterative experimentation together with users because they are the ones who can determine correctness. We present an empirical process model describing how data scientists practically elicited these"retrieval requirements"and managed system limitations. This work advances software engineering knowledge by providing insights into the specialized requirements engineering processes for implementing RAG systems in complex domain-specific applications.

Problem

• Exploring requirements engineering for RAG systems in expert settings
• Addressing tension between AI perfection expectations and output correctness
• Identifying context-specific retrieval requirements through iterative experimentation

Innovation

• Iterative experimentation with users for retrieval requirements
• Empirical process model for eliciting retrieval requirements
• Managing system limitations in domain-specific RAG applications