Louder, Longer, Livelier: Acoustic Shortcuts and Underspecified Rationales in Speech LLM Judges

2026-09-29

AI Summary

This study addresses the vulnerability of speech large language model evaluators to acoustic shortcuts, such as volume variations, and their tendency to generate rationales lacking acoustic grounding. We systematically audit six mainstream evaluators through controlled experiments manipulating audio intensity, content, and emotion variables. To more precisely expose preference biases, we propose pairwise comparison as an alternative to pointwise scoring, validated through human preference calibration and multimodal techniques. This work is the first to formally define and quantify the “acoustic shortcut” phenomenon, demonstrating that current evaluators blindly reward high-decibel inputs and exhibit constrained explanatory vocabularies. Furthermore, we open-source the SpeechJudgeAudit toolkit to facilitate research reproducibility.

Abstract

LLM-as-a-judge is widely used for evaluating text, but extending this paradigm to speech requires models to interpret acoustic as well as linguistic evidence. This introduces a modality-specific risk: a speech judge may treat a perceptually salient cue as evidence of quality even when that cue is irrelevant to the target criterion or receives more weight than human listeners give it. We call this behavior an acoustic shortcut. To study it, we audit six speech LLM judges using controlled manipulations of intensity, content richness, and emotional delivery. We evaluate both pointwise scoring and pairwise comparison, using human preference calibration to interpret the results. The judges consistently reward louder audio, prefer content-rich speech more strongly than human listeners do, and map emotional delivery into quality preferences. These effects are most visible in pairwise comparison, while pointwise scores often obscure them. More concerningly, the accompanying rationales rarely identify the acoustic cue that changes a judgment and instead repeatedly rely on a limited vocabulary, leaving them acoustically underspecified. Together, these findings show that reliable speech judges must both resist acoustic shortcuts and ground their rationales in the acoustic evidence behind their decisions. To support reproducibility and future audits, we also release SpeechJudgeAudit, the controlled stimuli and evaluation tools used in this study.

Problem

• Speech LLM Judges
• Acoustic Shortcuts
• LLM-as-a-judge
• Underspecified Rationales
• Speech Evaluation

Innovation

• Acoustic Shortcuts
• Speech LLM Judges
• LLM-as-a-judge
• Underspecified Rationales
• SpeechJudgeAudit