answer verification

This work addresses a key challenge in multi-hop retrieval-augmented generation (RAG): disentangling whether performance bottlenecks stem from missing supporting evidence or the reader’s inability to effectively utilize available evidence. To this end, the authors introduce the concept of an “evidence interface” and, for the first time, systematically decouple the impact of missing support chains from evidence interface effects by evaluating adapter-trained readers across three multi-hop QA benchmarks with annotated support chains. Using original contexts, retrieved passages, and oracle-supported inputs, they demonstrate that a support-supervised ranker significantly improves support coverage on 2Wiki and MuSiQue, recovers original-context performance at substantially lower prompting costs, and approaches the upper bound set by gold support annotations.

This work addresses the challenge that large language models often produce responses in high-stakes scenarios whose correctness is difficult for users to verify based on the provided reasoning. The paper introduces “error verifiability” as a novel quality dimension distinct from accuracy and proposes a balanced metric, \( v_{\text{bal}} \), to quantify how effectively a model’s rationale aids human judgment. To enhance verifiability, the authors develop two external-information-augmented rephrasing methods: Reflect-and-Rephrase (RR) for mathematical reasoning and Oracle-Rephrase (OR) for factual question answering. Experimental results demonstrate that conventional training and scaling approaches fail to improve verifiability, whereas RR and OR substantially increase human accuracy in judging answer correctness.

Existing QA benchmarks evaluate only answer correctness, neglecting plausible yet incorrect candidates—leading to coarse-grained model evaluation and limited robustness analysis. To address this, we propose PlausibleQA, the first large-scale, fine-grained QA benchmark explicitly designed to assess answer plausibility. It comprises 10K questions, 100K candidate answers, and 900K human-annotated pairwise plausibility comparisons. Crucially, PlausibleQA introduces continuous plausibility scores and multi-dimensional attribution explanations—novel features enabling nuanced plausibility modeling. Built via rigorous crowdsourced collaborative annotation and empirically validated, the benchmark establishes a new paradigm for plausibility-aware multiple-choice QA modeling and robustness evaluation. Experiments demonstrate its superiority over baselines in evaluating model discrimination capability and assessing distractor quality improvements. All data and annotations are publicly released.

This work addresses the limitation of existing Retrieval-Augmented Generation (RAG) evaluation methods, which predominantly focus on final answer quality and lack fine-grained assessment of individual evidence contributions during reasoning. The authors propose CUE-R, a novel framework that introduces lightweight intervention mechanisms—REMOVE, REPLACE, and DUPLICATE—to quantify the operational utility of evidence along four dimensions: correctness, proxy faithfulness, confidence error, and behavioral trajectory shift. This approach moves beyond answer-centric evaluation paradigms and reveals non-additive interaction effects among multi-hop evidence. Experiments on HotpotQA and 2WikiMultihopQA demonstrate that REMOVE and REPLACE significantly degrade performance and induce behavioral shifts, while DUPLICATE, though redundant, is not neutral; moreover, jointly removing multi-hop evidence causes performance degradation far exceeding that of single-evidence removal.

This work addresses the challenge that complex queries often admit multiple valid answers, which existing retrieval methods struggle to comprehensively cover. To this end, the authors propose the RVR framework, which employs an iterative closed-loop mechanism of “retrieve–verify–retrieve” to dynamically enrich the original query with verified documents, thereby enabling effective query expansion and answer discovery—without relying on complex agent-based systems. The approach requires only off-the-shelf or minimally fine-tuned retrievers and verifiers, yet achieves substantial gains in multi-answer recall. On the QAMPARI dataset, it yields a relative improvement of over 10% (3% absolute) in full-recall performance and consistently outperforms strong baselines across diverse domains, including QUEST and WebQuestionsSP.

This work addresses the challenge in retrieval-augmented generation (RAG) where retrieved passages are topically relevant yet lack sufficient evidential support for accurate answer generation. To tackle this, the authors propose SURE-RAG, a novel framework that models evidence sufficiency as a set-level property for the first time. The approach employs a shared pairwise claim-evidence verifier to produce local relational distributions, which are then aggregated into interpretable global signals—such as coverage, conflict, and uncertainty—and integrated with a calibration mechanism and a three-way decision strategy to generate answers only when evidence is deemed sufficient. Evaluated on HotpotQA-RAG v3, SURE-RAG achieves a Macro-F1 score of 0.9075, significantly outperforming baseline methods, reducing risk by 37%, and demonstrating full auditability while surpassing GPT-4o in controlled verification tasks.

This study addresses the instability and poor interpretability of short-answer visual question answering (VQA) benchmark evaluations, which often misclassify semantically correct answers as errors due to overreliance on superficial string matching. Leveraging a high-precision (97.6%) human-validated semantic judgment protocol, the authors conduct a systematic audit of over 37k official errors from six multimodal models across six benchmarks, revealing that nearly half of these “errors” are in fact semantically accurate but differ in surface form. Through text-only model replication, deterministic CPU-based repair contracts, and answer-type diagnostics, the work demonstrates that evaluation bias stems from the scorer’s preference for lexical form over meaning, and shows that simple prompting or contextual fine-tuning can substantially improve scoring stability. The study advocates semantic auditing and answer-type analysis as essential complements to standard VQA benchmark evaluation.

This work addresses the challenge of hidden misconceptions in student problem-solving, where learners often arrive at correct answers through flawed reasoning—rendering such errors invisible to conventional automated feedback systems. To tackle this issue, the authors propose a hierarchical assessment framework that decouples answer correctness from the validity of the solution process. They design a “detect–verify–escalate” pipeline that, under conditions of high uncertainty, triggers diagnostic follow-up questions rather than immediate alerts. Integrating a fine-tuned classifier with an open-source reasoning model and grounding the approach in educational theory, the system achieves low false-positive rates while maintaining high coverage of misconception detection. Empirical results demonstrate that the framework identifies 84% of hidden misconceptions, substantially reducing teacher intervention load and supporting dual deployment modes—teacher review and autonomous tutoring.

This work addresses the challenges of non-unique solutions and noise-induced temporary infeasibility in structured ranking and selection problems by proposing a unified framework, ENDS. The framework integrates answer-level acceptance sets, a constrained generalized likelihood ratio stopping rule, and a novel answer–trap decomposition mechanism, yielding a max-max-min eigenvalue characterization and a general information-directed sampling principle. By dynamically constructing acceptance sets, explicitly detecting traps, and incorporating cost-aware sampling, ENDS is broadly applicable to diverse settings such as multi-fidelity ranking and Condorcet winner identification. Empirical results demonstrate that the method achieves superior performance across a range of pure exploration tasks, confirming its generality and practical utility.

Current large language models often rely on internal memorization or dataset-specific shortcuts in multi-hop question answering, lacking genuine context-based reasoning capabilities. To address this limitation, this work proposes the CRiT-QA dataset, which innovatively integrates counterfactual entity substitution with multi-anchor distractor chains to construct a context-constrained evaluation framework for multi-hop reasoning. This design compels models to perform complete inference strictly based on provided evidence, thereby effectively suppressing shallow heuristic strategies. Empirical evaluations on mainstream large language models reveal significant vulnerabilities in their reasoning under counterfactual conditions and distractor traps, highlighting the dataset’s utility in enabling more reliable assessment of true multi-hop reasoning abilities.