🤖 AI Summary
Existing evaluation metrics struggle to accurately assess whether long-context or retrieval-augmented language models genuinely utilize evidence, often conflating memorization, superficial citation, and substantive integration. This work proposes a four-condition diagnostic protocol—comprising no-evidence, full-context, retrieved-evidence, and oracle-evidence settings—to disentangle evidence utilization into four diagnosable dimensions for the first time. It introduces the ONCU metric to quantify the degree to which models recover the performance advantage afforded by oracle evidence. Through carefully matched experimental designs, effective denominator group filtering, and multidimensional audit metrics, the study evaluates five families of open-source models across 18,000 predictions. Results reveal that synthetic tasks are primarily limited by failures in leveraging full context, whereas real multi-hop tasks suffer from insufficient retrieval chain coverage—a distinction consistently corroborated by ONCU scores.
📝 Abstract
Final-answer accuracy, retrieval recall, and citation overlap do not by themselves identify whether a long-context or retrieval-augmented language model used the evidence it was given. A model can answer from parametric memory, fail despite receiving the right passages, or cite evidence without converting it into the requested answer. This paper proposes a matched four-condition evidence-availability protocol--no evidence, full context, retrieved evidence, and oracle-evidence reference--for diagnosing evidence utilization under fixed examples, prompts, score fields, retrieval settings, and validity checks. ONCU is used as a protocol-bound estimator of recovered oracle-reference evidence advantage and is computed only for denominator-valid groups; denominator-free answer, evidence, retrieval, and failure-audit metrics are reported separately. The empirical study evaluates five local open-weight models from the Qwen, Gemma, Llama, and Mistral families across Controlled-ONCU-safe16K, HotpotQA-ONCU, and 2WikiMultiHopQA-ONCU, with 18,000 ONCU-compatible predictions. The main finding is a task-dependent bottleneck split: controlled synthetic settings primarily expose full-context utilization failures, whereas the tested realistic multi-hop settings primarily expose retrieval-chain coverage failures in denominator-free answer and evidence metrics, with ONCU supporting the same direction on oracle-improving groups. The contribution is a diagnostic protocol for separating no-evidence answerability, oracle-evidence recoverability, full-context utilization, and retrieval-conditioned utilization, rather than a single-score leaderboard for long-context or retrieval-augmented systems.