MIRAGE: Agentic Framework for Multimodal Misinformation Detection with Web-Grounded Reasoning

2025-10-20

AI Summary

To address the limitations of supervised multimodal fake news detection models—namely, their reliance on domain-specific annotations and poor generalization—this paper proposes a zero-shot, agent-based framework for inference-time reasoning. The framework decomposes detection into four modular components: visual authenticity assessment, cross-modal consistency analysis, retrieval-augmented fact-checking, and calibrated judgment synthesis. It integrates vision-language models, iterative web retrieval, retrieval-augmented generation (RAG), and multi-signal fusion to produce traceable, structured reasoning outputs. Evaluated on a 1,000-sample validation set, it achieves 81.65% F1 and 75.1% accuracy (with a 34.3% false positive rate); on a 5,000-sample test set, F1 remains stable at 81.44%, significantly outperforming strong baselines. The core contribution is the first modular, zero-shot multimodal verification paradigm that eliminates domain-specific training while ensuring robust performance and cross-strategy generalization.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Misinformation spreads across web platforms through billions of daily multimodal posts that combine text and images, overwhelming manual fact-checking capacity. Supervised detection models require domain-specific training data and fail to generalize across diverse manipulation tactics. We present MIRAGE, an inference-time, model-pluggable agentic framework that decomposes multimodal verification into four sequential modules: visual veracity assessment detects AI-generated images, cross-modal consistency analysis identifies out-of-context repurposing, retrieval-augmented factual checking grounds claims in web evidence through iterative question generation, and a calibrated judgment module integrates all signals. MIRAGE orchestrates vision-language model reasoning with targeted web retrieval, outputs structured and citation-linked rationales. On MMFakeBench validation set (1,000 samples), MIRAGE with GPT-4o-mini achieves 81.65% F1 and 75.1% accuracy, outperforming the strongest zero-shot baseline (GPT-4V with MMD-Agent at 74.0% F1) by 7.65 points while maintaining 34.3% false positive rate versus 97.3% for a judge-only baseline. Test set results (5,000 samples) confirm generalization with 81.44% F1 and 75.08% accuracy. Ablation studies show visual verification contributes 5.18 F1 points and retrieval-augmented reasoning contributes 2.97 points. Our results demonstrate that decomposed agentic reasoning with web retrieval can match supervised detector performance without domain-specific training, enabling misinformation detection across modalities where labeled data remains scarce.

Problem

• Detecting multimodal misinformation combining text and images across web platforms
• Overcoming limitations of supervised models requiring domain-specific training data
• Verifying claims through web-grounded reasoning when labeled data is scarce

Innovation

• Decomposes verification into four sequential reasoning modules
• Integrates vision-language models with targeted web retrieval
• Uses model-pluggable framework without domain-specific training