When Should Active RAG Retrieve? A Budget-Aware Evaluation of Utility, Calibration, and Cost

2026-07-27

AI Summary

This work addresses the lack of a well-defined retrieval budget in existing Active Retrieval-Augmented Generation (Active RAG) systems, which distorts performance comparisons. The authors formalize active retrieval as a utility estimation problem, quantifying retrieval value through marginal changes in correctness, and introduce the first budget-aware evaluation paradigm. Their framework decouples utility ranking, threshold calibration, and deployment cost via novel metrics—including the utility frontier, threshold transfer error, and harm auditing. Empirical results reveal that retrieval-induced harms are non-negligible and that simple baselines often outperform learned routers. The study advocates for comprehensive reporting of frontier curves, actual retrieval usage rates, and cost breakdowns to accurately reflect method robustness across datasets and budget constraints.

Abstract

Active RAG systems decide when to retrieve external knowledge during generation, making them a budget-sensitive case of agentic RAG and self-adaptive retrieval. Yet evaluations often leave the operating point underspecified: two systems may both claim a 50% evidence-usage budget while realizing different held-out usage rates, so higher accuracy can reflect a looser budget rather than a better retrieval policy. We study budget-aware evaluation for Active RAG by recasting active retrieval as utility estimation, where retrieval is valuable only through its marginal correctness change over a no-retrieval answer. This view separates three questions that single-point evaluations conflate: whether trigger scores rank useful retrieval decisions, whether thresholds calibrated on past data meet future budgets, and how trigger-side computation changes deployment cost. We operationalize these questions with exact top-k utility frontiers, deployable threshold frontiers, conservative budget frontiers, harm audits, and cost decompositions. Across knowledge-intensive multi-hop QA datasets and open instruction models, retrieval harm is non-negligible, router rankings change across datasets and budgets, nominal thresholds can miss target usage, and simple uncertainty or retrieval-score baselines often rival learned utility routers. Budget-aware Active RAG evaluations should therefore report frontiers, realized usage, threshold-transfer error, harm rates, and cost decompositions alongside accuracy.

Problem

• Active RAG
• budget-aware evaluation
• retrieval utility
• calibration
• cost decomposition

Innovation

• Active RAG
• budget-aware evaluation
• utility estimation
• retrieval harm
• threshold calibration