MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models

2026-09-27

AI Summary

This study addresses the challenge that existing benchmarks struggle to disentangle hallucinations from comprehension failures in large audio models. To bridge this gap, this work proposes the first evaluation benchmark encompassing both contextual and knowledge-based hallucinations, comprising 12,000 data pairs, alongside a comprehensive evaluation pipeline. Specifically, it formalizes two categories of hallucinations and introduces a reference-based groundedness evaluator. The pipeline further integrates open-ended question answering with human-annotation-guided LLM judges for multi-domain assessment. Extensive experiments reveal that state-of-the-art models exhibit hallucination rates as high as 36.5%, indicating that effectively mitigating audio hallucinations remains an urgent and unresolved open challenge.

Abstract

Large audio-language models (LALMs) produce fluent responses about audio but often hallucinate by making plausible yet ungrounded claims. Existing audio hallucination benchmarks mainly measure response correctness, leaving it unclear whether an LALM hallucinates or simply fails to understand the audio. We challenge correctness-based evaluation by defining two hallucination categories: (i) context, where claims are not grounded in the audio; and (ii) knowledge, where claims about audio-related topics lack support from externally verifiable facts. We introduce MISHAP-Bench, a comprehensive benchmark with 12,000 challenging open-ended question-audio pairs and a rigorous evaluation pipeline covering both categories. To evaluate open-ended responses, we propose a groundedness judge that uses reference rubrics and judge prompts guided by human annotations. We extensively evaluate ten state-of-the-art LALMs and show that hallucination remains substantial. Even a frontier model such as Gemini 3.7 Flash reaches a hallucination rate of 36.5%. We further adapt and benchmark four mitigation methods from multiple domains for LALMs. Despite some improvements, effective hallucination mitigation remains an open challenge. Finally, we call on the community to evaluate hallucination and benchmark mitigation methods with MISHAP-Bench.

Problem

• Hallucination
• Large Audio-Language Models
• Benchmark
• Groundedness
• Evaluation

Innovation

• Hallucination Benchmark
• Large Audio-Language Models
• Groundedness Judge
• Context and Knowledge Hallucination
• Open-ended Evaluation