Learning Emotion-Invariant Speaker Representations for Speaker Verification

2024-04-14

IEEE International Conference on Acoustics, Speech, and Signal Processing

AI Summary

To address speaker representation degradation and verification performance decline caused by emotional variability, this paper proposes an emotion-robust speaker representation learning framework. Methodologically, it introduces three novel components: (1) Copy-Paste speech data augmentation, (2) a cosine similarity–constrained loss function, and (3) an energy-based masking (EM) mechanism for emotion-aware spectral suppression. The EM module dynamically attenuates emotion-discriminative frequency bands using frame-level speech energy, thereby explicitly disentangling speaker identity from emotion-related variations and enhancing representation invariance. Evaluated on standard benchmarks, the proposed approach achieves a 19.29% relative reduction in equal error rate (EER) over baseline systems. Ablation studies confirm the individual efficacy and synergistic benefits of all components. This work provides a principled, reproducible methodology for building highly robust speaker verification systems under emotional variability.

Technology Category

Natural Language Processing: Safety and RobustnessIntelligent Robots: Multimodal Perception & Sensor FusionMachine Learning: Representation Learning

Application Category

Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web searchUser Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systemsWeb Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

In recent years, the rapid progress in speaker verification (SV) technology has been driven by the extraction of speaker representations based on deep learning. However, such representations are still vulnerable to emotion variability. To address this issue, we propose multiple improvements to train speaker encoders to increase emotion robustness. Firstly, we utilize CopyPaste-based data augmentation to gather additional parallel data, which includes different emotional expressions from the same speaker. Secondly, we apply cosine similarity loss to restrict parallel sample pairs and minimize intraclass variation of speaker representations to reduce their correlation with emotional information. Finally, we use emotion-aware masking (EM) based on the speech signal energy on the input parallel samples to further strengthen the speaker representation and make it emotion-invariant. We conduct a comprehensive ablation study to demonstrate the effectiveness of these various components. Experimental results show that our proposed method achieves a relative 19.29% drop in EER compared to the baseline system.

Problem

• Enhancing speaker verification robustness against emotional variability
• Reducing emotion correlation in speaker representations
• Improving emotion-invariant features via data augmentation and masking

Innovation

• CopyPaste data augmentation for parallel emotional data
• Cosine similarity loss to reduce emotional correlation
• Emotion-aware masking to strengthen speaker representation