A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility

2026-07-27

AI Summary

This study addresses the challenge of inefficient retrieval of multi-source, heterogeneous knowledge in the operation of large-scale scientific facilities, which hinders operational efficiency. The authors propose APS-RAG, a retrieval-augmented generation platform that integrates dense, sparse, and knowledge graph-based retrieval channels, enhanced by query-type-adaptive reciprocal rank fusion, cross-encoder reranking, and a ReAct agent execution framework grounded in the MCP tool protocol. Key contributions include the creation of APS-Bench—the first operation-oriented evaluation benchmark for large scientific facilities—the open-sourcing of a six-layer evaluation framework and an agent skill taxonomy, and empirical validation showing that the full system achieves a 70.3% key information recall on APS-Bench, significantly outperforming the BM25 baseline (63.8%), with ablation studies confirming the critical role of the cross-encoder in performance gains.

Abstract

Scientific user facilities accumulate decades of operational knowledge that no single search index covers: electronic logbooks, technical documents, internal wikis, operations chat messages, maintenance records, and live control-system data. We present APS-RAG, Advanced Photon Source Retrieval Augmented Generation, a deployed platform that makes the institutional knowledge at the Advanced Photon Source (APS) accessible to staff through natural-language queries, along with an operations-grounded evaluation. The retrieval engine fuses dense, sparse, and knowledge-graph (KG) channels with query-type-adaptive reciprocal-rank fusion, adds a corrective agentic loop, and runs a native-tool ReAct executor over a Model Context Protocol (MCP) tooling layer. We construct APS-Bench, a 50-question, question-answering (QA) dataset with auditable gold answers. Every retrieval-augmented variant numerically improves strict vital-nugget recall over a naive BM25 baseline (63.8%), with the full corrective Agentic GraphRAG scoring (70.3%). The cross-encoder reranker contributes significantly to answer quality: removing it and allowing the LLM to score relevance drastically reduces strict vital recall by 32.8%. The graph channel and corrective loop contribute positively as expected, but the performance gains are marginal. Additionally, we also compare the performance of open-source and closed-source LLMs in final answer synthesis. We release the APS-Bench construction methodology, the six-layer evaluation harness, and the underlying codebase, along with the '/aps-rag' retrieval agent skill framework, to support reproduction and adoption at other facilities. Together, the deployed platform and its operations-grounded evaluation present a promising workflow for trustworthy, statistically grounded AI assistance in facility operations, transferable to other large scientific instruments.

Problem

• scientific facility
• institutional knowledge
• retrieval-augmented generation
• natural-language query
• operational knowledge

Innovation

• Corrective Agentic RAG
• Operations-Grounded Evaluation
• Knowledge Graph Fusion
• Model Context Protocol (MCP)
• APS-Bench