Your Multimodal Speech Model Says I Have a Face for Radio

2026-05-28

AI Summary

This study investigates whether incorporating visual information into multimodal speech recognition models exacerbates gender and racial biases. To this end, the authors introduce a controlled evaluation framework that pairs identical audio clips with synthetic videos of faces varying in gender and race, enabling systematic assessment of transcription performance across demographic groups. Using this setup, they evaluate prominent multimodal large language models, including mWhisper-Flamingo and Gemini, and uncover significant disparities—up to 4.05 percentage points in word error rate (WER)—between different demographic subgroups. These findings demonstrate that the visual modality can introduce new fairness risks in multimodal systems, offering critical empirical evidence to inform future bias evaluation and mitigation strategies in this domain.

Abstract

As large neural models have become better at language tasks, researchers are increasingly building multi- and omnimodal models that handle more modalities of data. One example is the expansion of speech recognition models to audio-visual data for noise mitigation and multimodal subtitling. While performance and bias have been studied extensively in the single-modality regime, it is unknown how new modalities affect this, even though they produce biases in humans. We therefore propose the first bias evaluation of multimodal speech recognition, where we create videos pairing different faces with the same audio, and measure changes in speech transcription accuracy. We find large quality-of-service differences across mWhisper-Flamingo and Gemini models, with drops of up to 4.05 word error rate points, across self-declared gender, ethnicity, and their intersection. Our findings point to a priority for developers to evaluate, fix, and communicate such limitations, as providing more signals through additional modalities is not necessarily better, and may even lead to biased outcomes.

Problem

• multimodal speech recognition
• bias
• fairness
• audio-visual models
• word error rate

Innovation

• multimodal speech recognition
• bias evaluation
• audio-visual modeling
• fairness in AI
• word error rate