Beyond Self-Knowledge: Propagating Uncertainty Across Reasoning and Retrieval in LLMs

2026-07-28

AI Summary

This work addresses the limitations of existing retrieval-augmented generation approaches, which often retrieve irrelevant information indiscriminately—increasing computational overhead—and lack effective mechanisms to assess retrieval necessity. The authors propose an uncertainty-guided dynamic retrieval framework that prompts large language models to produce structured preliminary answers along with verbalized confidence scores. Using a validation set, a model-specific confidence threshold is established; TF-IDF retrieval is triggered only when confidence falls below this threshold. The method integrates a two-stage question-answering pipeline with a post-hoc verification strategy. Evaluated across six question-answering benchmarks, it achieves an average token-level F1 score of 0.483—significantly outperforming both always-retrieve (0.467) and never-retrieve (0.401) baselines—while reducing retrieval calls by 20.4%. It also surpasses random routing in 17 out of 18 experimental settings, demonstrating that uncertainty serves as an effective and efficient signal for retrieval routing.

Abstract

Retrieval-augmented generation improves knowledge-intensive question answering, but indiscriminate retrieval can introduce irrelevant evidence and unnecessary computation. We investigate whether verbalized confidence from black-box language models can serve as an actionable signal for retrieval routing. Our method, BeyondUncertainty, first elicits a structured provisional answer and confidence estimate, then applies a model-specific threshold selected on held-out validation data and frozen before test evaluation. Low-confidence questions receive top-$5$ TF--IDF retrieval followed by a second answer call, whereas high-confidence questions return the provisional answer directly. We evaluate 27,000 policy instances across six QA benchmarks, three model families, and three retrieval policies. BeyondUncertainty achieves $0.483$ mean token-level F1, compared with $0.467$ for always retrieval and $0.401$ for no retrieval, while reducing retrieved passages by $20.4\%$ relative to always retrieval. When matched on the number of questions routed to retrieval within each dataset--model cell, it outperforms a post-hoc random allocation in 17 of 18 settings, with an average gain of 0.024 F1. Although poorly calibrated as an absolute probability, probe uncertainty modestly predicts question-level retrieval benefit (AUROC $=0.628$). However, the additional probe increases total token usage by $28.2\%$, revealing a trade-off between more selective evidence acquisition and end-to-end token efficiency.

Problem

• retrieval-augmented generation
• uncertainty propagation
• retrieval routing
• large language models
• confidence estimation

Innovation

• retrieval-augmented generation
• uncertainty propagation
• confidence-based routing
• black-box LLMs
• selective retrieval