Inference-Time Agentic Decision Rules Beat Longer Evolving Search for Multi-Image Medical Reasoning

2026-07-29

AI Summary

This study addresses the challenges of multi-image medical visual question answering, where models must effectively aggregate cross-image evidence, remain robust to answer-order perturbations, and avoid overfitting to noisy feedback during inference. Within a unified high-budget evolutionary framework (ShinkaEvolve), the authors systematically evaluate five inference-time agent decision strategies. Surprisingly, the simplest approach—sequential voting (order-vote)—significantly outperforms more complex re-ranking methods and variants that extend evolutionary search. On a frozen test set, order-vote achieves an accuracy of 57.89 ± 0.65%, substantially exceeding both the baseline (52.73 ± 0.42%) and re-ranking schemes (55.79 ± 0.43%). Bootstrap testing confirms the robustness of this improvement, revealing that straightforward aggregation mechanisms can offer superior performance and generalization in complex medical reasoning tasks.

Abstract

Multi-image medical VQA is not merely a prompt-length problem; it is a fundamental challenge of agentic decision-making. Medical vision-language agents must aggregate evidence across ordered images, remain robust to answer-order perturbations, and avoid overfitting to noisy search-time feedback. We study MedFrameQA through a controlled comparison of five inference-time agentic strategies, optimized using the same high-budget ShinkaEvolve configuration and evaluated on a reproducible internal frozen split (1,331 evolution, 665 holdout, 855 final test). Across five independent repeated runs, the strongest method emerges as the simplest robust aggregator: the \textbf{order-vote} policy achieves $57.89 \pm 0.65\%$ final-test accuracy, significantly outperforming the fixed baseline ($52.73 \pm 0.42\%$) and the more complex, albeit brittle, order-rerank variant ($55.79 \pm 0.43\%$). Paired bootstrap analysis confirms these significant gains. Extending the evolutionary search budget from 50 to 100 generations yields no generalization benefit: while holdout performance marginally increases, final-test accuracy drops from $57.89\%$ to $56.02\%$. Our findings suggest that for multi-image medical reasoning, defining the correct agentic decision rule is substantially more impactful than expanding the optimization search budget.

Problem

• multi-image medical VQA
• agentic decision-making
• evidence aggregation
• answer-order robustness
• noisy feedback overfitting

Innovation

• agentic decision rules
• multi-image medical VQA
• order-vote
• inference-time strategy
• evolutionary search