Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition

2026-06-05

AI Summary

This study investigates whether audio language models leverage explicit acoustic cues in a grounded manner for speech emotion recognition, even when raw audio is available. To this end, six interpretable acoustic concepts—such as energy, pitch, and voice quality—are derived from eGeMAPS features and embedded as symbolic tokens into textual prompts, while keeping the original audio input unchanged. Through perturbation analyses—including token alignment, shuffling, conflict, and corruption—the authors find that aligned tokens significantly improve unweighted average recall (UAR) on FAU-Aibo and IEMOCAP datasets, whereas perturbations degrade performance and bias predictions toward neutrality. Notably, the models retain partial reliance on raw audio under strong perturbations. This work introduces symbolic acoustic tokens as an interpretability probe, offering the first evidence of models’ sensitivity to and robustness with respect to explicit acoustic cues.

Abstract

Instruction-following audio language models (ALMs) can be augmented with explicit acoustic cues, yet it remains unclear whether such cues are used in a grounded way when the raw audio is already available. We study this question in speech emotion recognition (SER) by deriving six interpretable acoustic concept tokens from the standardised eGeMAPS paralinguistic feature set. These tokens summarise energy, pitch, dynamics, brightness, formants, and voice quality, and are appended to the textual prompt while the audio input is kept unchanged. Across the widely used FAU-Aibo and IEMOCAP benchmarks, aligned tokens improve unweighted average recall (UAR), whereas shuffled, conflicting, or corrupted tokens reduce performance relative to aligned tokens and shift confusions toward neutral. Importantly, predictions do not collapse under strong token perturbations, suggesting that the models are sensitive to the symbolic cue channel but remain partly anchored to the audio signal. We argue that token-only interventions provide a practical way to probe audio-grounded cue use, robustness, and interpretability in ALM-based affective computing.

Problem

• acoustic cue alignment
• speech emotion recognition
• audio language models
• affective computing
• grounded representation

Innovation

• acoustic cue alignment
• audio language models
• speech emotion recognition
• interpretable tokens
• eGeMAPS