To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech

2026-09-24

AI Summary

This study addresses the challenge of audio misinformation verification by revealing a significant text-speech modality gap in Large Audio-Language Models (LALMs) during fact-checking. To tackle this, we construct the VeriSpeak benchmark and propose a method integrating explicit reasoning with retrieved evidence contrast to mitigate model confusion. Experimental results demonstrate that naive retrieval augmentation yields only marginal gains, whereas an LALM fine-tuned with chain-of-thought reasoning, combined with evidence-grounded inference, achieves 86.1% accuracy. This work confirms the necessity of explicit reasoning for multimodal fact verification and establishes an effective paradigm for detecting speech-based misinformation.

Abstract

Online misinformation increasingly appears in spoken formats such as news clips, podcasts, interviews, political speeches, and social media videos, creating a need for fact-checking systems that can verify claims directly from speech. We introduce VeriSpeak, a probe benchmark for studying speech-based fact verification in Large Audio Language Models (LALMs). VeriSpeak contains 3,879 spoken claims spanning temporal, geographical, and relational facts, with balanced true and false labels. The benchmark is designed to examine whether factual verification ability transfers from text to speech, and whether retrieval-augmented LALMs can use textual evidence to correctly support or refute spoken claims. Our experiments reveal a consistent text-speech modality gap: LALMs that verify written claims reliably often fail on the same claims when spoken. Moreover, retrieval alone provides limited gains because models frequently conflate retrieved evidence with the spoken claim. In contrast, retrieval combined with explicit reasoning improves claim-evidence comparison, with a thinking-tuned LALM reaching 86.1% accuracy. VeriSpeak highlights that effective speech misinformation detection requires not only speech understanding, but also grounded reasoning over retrieved evidence. The dataset is publicly available via Hugging Face at https://huggingface.co/datasets/abhiram4572/VeriSpeak.

Problem

• speech fact-checking
• misinformation detection
• Large Audio Language Models
• modality gap
• retrieval-augmented verification

Innovation

• Fact Checking
• Large Audio Language Models
• Retrieval-Augmented Generation
• Speech Verification
• Reasoning