Best-of-Evidence: Best-of-N Selection under Partial Verification

2026-07-23

AI Summary

This work addresses the challenge in vision-language tasks where full responses are difficult to verify, yet only partial information—such as regions, relations, or numerical values—is verifiable, rendering traditional Best-of-N (BoN) selection ineffective. The authors propose the Best-of-Evidence (BoE) framework, which formalizes candidate selection under partial verification for the first time. BoE models reusable claims via a signed candidate-factor graph and dynamically selects the most decision-influential evidence within a limited query budget. Theoretically, shared factor queries reduce query complexity from Θ(K) to O(log K), with BoE naturally degenerating to BoN under zero budget. Experiments demonstrate that BoE significantly improves selection performance across four medical VQA datasets, rectifies failure cases of BoN, and reveals that channel quality and candidate generation capacity critically constrain achievable performance.

Abstract

BoN improves model outputs by sampling several candidates and selecting one with a proxy score, but it assumes that complete candidates can be evaluated reliably. Many vision-language tasks instead provide only partial verification: a finding, span, value, region, or relation may be checkable even when no dependable whole-response verifier exists. Moreover, the same claim may recur across candidates with opposing stances, allowing one observation to support part of the pool and contradict another. We introduce Best-of-Evidence (BoE), an inference-time selection framework that keeps the BoN candidate pool fixed, represents reusable claims with a signed candidate--factor graph, and allocates a limited budget to evidence actions that can change the final choice. BoE formalizes selection under partial verification and provides a practical score-based controller, with the zero-budget case recovering the underlying BoN decision. Theoretically, we show that residual evidence capacity limits any evidence-driven improvement and that shared factor queries can achieve an O(log K) versus Θ(K) query separation in a factor-code model. Common-ledger experiments on four medical VQA settings show that BoE can improve fixed-pool selection and rescue some BoN failures when evidence is reliable, contrastive, and decision-relevant, while also revealing the channel-quality and candidate-generation limits that prevent universal gains.

Problem

• partial verification
• vision-language tasks
• candidate selection
• evidence-based reasoning
• Best-of-N

Innovation

• Best-of-Evidence
• partial verification
• factor graph
• evidence budgeting
• vision-language reasoning