Constructing and Evaluating Declarative RAG Pipelines in PyTerrier

2025-06-12

AI Summary

This paper addresses the complexity and poor reproducibility in building Retrieval-Augmented Generation (RAG) systems. To this end, it introduces the first declarative RAG framework natively built on PyTerrier’s architecture. Methodologically, it proposes a unified declarative operator syntax that enables end-to-end integration of indexing, sparse/dense retrieval, neural re-ranking, and LLM-based generation—supporting composable, reproducible, and evaluable pipelines. Its key contribution is the first native support for the full RAG workflow within PyTerrier, seamlessly leveraging its state-of-the-art retrieval and ranking components. Experiments on standard benchmarks—including Natural Questions—demonstrate that the framework significantly lowers the barrier to RAG development while preserving efficiency and rigorous evaluability. By enabling fully reproducible experimentation and systematic RAG research, it establishes a new paradigm for transparent, modular, and benchmark-driven RAG system design.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Search engines often follow a pipeline architecture, where complex but effective reranking components are used to refine the results of an initial retrieval. Retrieval augmented generation (RAG) is an exciting application of the pipeline architecture, where the final component generates a coherent answer for the users from the retrieved documents. In this demo paper, we describe how such RAG pipelines can be formulated in the declarative PyTerrier architecture, and the advantages of doing so. Our PyTerrier-RAG extension for PyTerrier provides easy access to standard RAG datasets and evaluation measures, state-of-the-art LLM readers, and using PyTerrier's unique operator notation, easy-to-build pipelines. We demonstrate the succinctness of indexing and RAG pipelines on standard datasets (including Natural Questions) and how to build on the larger PyTerrier ecosystem with state-of-the-art sparse, learned-sparse, and dense retrievers, and other neural rankers.

Problem

• Formulating RAG pipelines declaratively in PyTerrier
• Providing easy access to RAG datasets and evaluation
• Demonstrating succinct indexing with state-of-the-art retrievers

Innovation

• Declarative PyTerrier architecture for RAG pipelines
• PyTerrier-RAG extension with standard datasets and LLMs
• Integration with sparse and dense retrievers in PyTerrier