🤖 AI Summary
In speech emotion recognition, severe temporal misalignment between ASR transcripts and speaker diarization (SD) timestamps critically undermines the reliability of multimodal systems in conversational settings. To address this, we propose an end-to-end timestamp alignment pipeline that dynamically calibrates temporal boundaries between ASR and SD outputs, integrating cross-attention fusion and a gating mechanism to achieve precise inter-modal synchronization. The method leverages pre-trained Wav2Vec (audio), RoBERTa (text), and an SD model, requiring no additional annotations for fine-grained temporal alignment. Experiments on IEMOCAP demonstrate a 3.2% absolute improvement in emotion classification accuracy over unaligned baselines. This work constitutes the first systematic empirical validation of the critical role of temporal synchronization in multimodal emotion analysis, establishing a new benchmark for robust, time-aware multimodal modeling in dialogue scenarios.
📝 Abstract
In this paper, we investigate the impact of incorporating timestamp-based alignment between Automatic Speech Recognition (ASR) transcripts and Speaker Diarization (SD) outputs on Speech Emotion Recognition (SER) accuracy. Misalignment between these two modalities often reduces the reliability of multimodal emotion recognition systems, particularly in conversational contexts. To address this issue, we introduce an alignment pipeline utilizing pre-trained ASR and speaker diarization models, systematically synchronizing timestamps to generate accurately labeled speaker segments. Our multimodal approach combines textual embeddings extracted via RoBERTa with audio embeddings from Wav2Vec, leveraging cross-attention fusion enhanced by a gating mechanism. Experimental evaluations on the IEMOCAP benchmark dataset demonstrate that precise timestamp alignment improves SER accuracy, outperforming baseline methods that lack synchronization. The results highlight the critical importance of temporal alignment, demonstrating its effectiveness in enhancing overall emotion recognition accuracy and providing a foundation for robust multimodal emotion analysis.