Silent Failures in Multimodal Agentic Search:A Diagnostic Taxonomy and Cross-Judge Evaluation

2026-07-22

AI Summary

This work addresses the prevalent yet overlooked issue of “silent failures” in multimodal agents tackling knowledge-intensive visual question answering—cases where final answers appear correct despite flawed reasoning trajectories. Conventional evaluation methods focusing solely on answer accuracy fail to detect such errors. To this end, the study introduces the first systematic taxonomy encompassing six distinct types of silent failures and proposes a trajectory-level evaluation framework grounded in the ReAct paradigm. Novel evaluation paradigms—including cross-examiner validation, blank-image stress testing, and tool ablation—are incorporated to expose hidden reasoning flaws. Empirical assessment of four state-of-the-art models on the MMSearch-Plus benchmark reveals that surface-level accuracy substantially overestimates true reasoning reliability, and that silent failures evolve or shift rather than disappear as model capabilities advance.

Abstract

Multimodal agentic search systems increasingly rely on external tools to answer knowledge-intensive visual questions. However, existing evaluations mainly focus on final-answer accuracy and may miss failures in the search trajectory. In this work, we study such hidden reliability issues as silent failures. We introduce a six-category taxonomy covering modality shortcuts, phantom grounding, wrong-evidence-right-answer cases, over-retrieval laundering, cross-modal contradiction, and provenance hallucination. Based on this taxonomy, we build a trajectory-level diagnostic pipeline that evaluates both answer correctness and evidence-grounding quality under a unified ReAct-style scaffold. Experiments on MMSearch-Plus trajectories across four frontier multimodal models show that surface accuracy consistently overestimates true trajectory-level correctness. We further use cross-judge validation, blank-image stress tests, and tool ablations to show that silent failures are capability-dependent and often shift rather than disappear. Home-page: https://github.com/DingWu1021/silent-failures-multimodal-agentic-search

Problem

• silent failures
• multimodal agentic search
• trajectory-level evaluation
• evidence grounding
• reliability assessment

Innovation

• silent failures
• multimodal agentic search
• diagnostic taxonomy
• trajectory-level evaluation
• evidence grounding