Reading Emotions in the Token Space: Discriminative Adaptation of SpeechLLMs for Emotion Recognition

2026-09-17

AI Summary

为解决SpeechLLMs在情绪识别中的分类问题，提出了一种通过分类头读取最终提示符隐藏状态的判别适应方法，提高了Macro F1并减少了幻觉。

Abstract

SpeechLLMs have shown strong potential for emotion recognition, yet they read the predicted emotion off a generative decoder not suited for classification: it can emit labels outside the target set and favors frequent classes. We propose a discriminative adaptation that reads the final prompt token's hidden state through a classification head, producing a label in one forward pass without modifying the backbone. Because this readout starts from the hidden state the model would otherwise decode, it gives a controlled comparison of generative and discriminative inference in an otherwise identical speechLLM. We keep the head a single linear layer, trading little accuracy for interpretability: each emotion becomes one direction in the LLM output token space, revealing associated tokens. On IEMOCAP, across two speechLLM architectures, it improves Macro F1 and removes hallucinations, with largest gains on realistic ASR transcripts. Our analysis reveals that these emotion directions encode indirect associations mirroring biases in web-scale text.

Problem

• SpeechLLMs
• emotion recognition
• generative decoder
• classification

Innovation

• discriminative adaptation
• classification head
• hidden state
• emotional directions
• interpretability