embedding extraction

Traditional monolithic speech embeddings conflate multiple attributes—such as linguistic content, speaker identity, dialect, and gender—hindering fine-grained similarity measurement. This work proposes a Factorized Embedding Framework that maps speech into a single embedding whose subspaces explicitly disentangle distinct phonetic attributes. The framework employs a shared acoustic encoder coupled with attribute-specific linear projection heads for each variation axis. Training leverages either knowledge distillation or contrastive learning, augmented by a signed axis-weighting mechanism to flexibly amplify or suppress targeted attributes. To the best of our knowledge, this is the first approach to achieve explicit multi-attribute disentanglement within a unified embedding space. Experiments demonstrate substantial improvements in recall on cross-corpus semantic retrieval tasks and effective mitigation of speaker bias, thereby validating the efficacy of multi-axis controllable similarity metrics.

This study systematically investigates the intrinsic semantic and syntactic properties of mainstream word embedding methods—such as Word2Vec and GloVe—and their performance disparities across diverse natural language processing tasks. By establishing a unified evaluation framework that integrates publicly available pretrained embeddings with standard benchmark datasets, the work conducts empirical comparisons on canonical tasks including semantic similarity and analogical reasoning. The findings delineate the performance boundaries and optimal application scenarios for each embedding model, offering practitioners reliable guidance for model selection in real-world settings. Furthermore, the analysis deepens the understanding of the inherent limitations of static word representations, highlighting critical constraints in capturing contextual and compositional linguistic phenomena.

This work investigates the cross-modal fusion capability of spoken language models (SLMs) under speech-text sentiment inconsistency. To address the limitation that existing SLMs over-rely on textual semantics while neglecting acoustic sentiment cues, we propose a novel sentiment-inconsistent speech evaluation paradigm and introduce EMIS—the first controllable synthetic dataset comprising speech samples with conflicting semantic and prosodic sentiment. Leveraging four state-of-the-art SLMs, we conduct cross-modal attention analysis and ablation studies. Results demonstrate that current SLMs predominantly base sentiment predictions on text, with minimal contribution from acoustic features, revealing a severe modality imbalance in their cross-modal fusion mechanisms. This study is the first to systematically expose such modality bias in SLMs’ sentiment understanding. We publicly release the EMIS dataset and associated code to establish a benchmark and guide future development of robust multimodal sentiment models.

Existing emotional TTS systems are constrained by discrete emotion labels and sparse annotation, limiting their ability to capture the continuity and complexity of human affect. This paper proposes the first method to seamlessly integrate the psychological PAD (Pleasure-Arousal-Dominance) three-dimensional emotion model into a language-model-driven TTS framework—enabling unsupervised disentanglement and learning of continuous emotional styles directly from expressive speech, without requiring explicit emotion labels. Key innovations include: (1) a classification-based emotion dimension predictor trained on labeled speech data, and (2) an end-to-end LM-TTS architecture jointly modeling linguistic and psychometric representations. Experiments demonstrate that our approach significantly improves emotional naturalness and spectral coverage of synthesized speech under zero-shot emotion-label supervision. Both objective metrics (e.g., F0 variance, spectral contrast) and subjective MOS scores surpass those of state-of-the-art baselines.

This study investigates whether individual dimensions in the representations of self-supervised speech models (specifically WavLM) encode distinct speaker-related acoustic attributes, such as pitch, gender, intensity, noise level, and the second formant. By applying principal component analysis (PCA) to disentangle model features, the authors systematically identify independent dimensions that exhibit strong correlations with these acoustic properties, establishing for the first time a clear correspondence between specific latent dimensions and interpretable speaker characteristics. Further experiments demonstrate that manipulating these dominant dimensions enables effective control over the associated speaker attributes in speech synthesis, thereby confirming both their controllability and practical utility in downstream applications.

To address poor data quality and low-frequency emotion class recognition in Multimodal Emotion Recognition in Conversations (MERC), this paper proposes a trimodal fusion method integrating speaker-identity-aware transfer learning with the MAMBA architecture. First, we establish a systematic data quality validation pipeline. Second, we extract audio-visual speaker identity embeddings using RecoMadeEasy® and model textual sentiment representations via MPNet-v2; an emotion-specific MLP collaborates with MAMBA to capture dynamic cross-modal dependencies. Third, speaker and facial identity features are explicitly transferred to model individual differences in emotional expression, significantly enhancing discriminability for sparse emotion classes. Evaluated on MELD and IEMOCAP, our method achieves 64.8% and 74.3% accuracy, respectively—outperforming state-of-the-art approaches. Key contributions include: (1) a principled data quality assessment framework; (2) identity-aware multimodal representation learning with MAMBA-based dynamic fusion; and (3) improved generalization to infrequent emotion categories through speaker-identity transfer.

This work addresses emotion recognition in conversational scenarios by effectively integrating multimodal information to enhance performance. We propose a lightweight multimodal baseline system that combines a Transformer-based text classifier with a self-supervised speech representation model, employing a simple late-fusion strategy for emotion prediction. Experimental results on the SemEval-2024 Task 3 dataset demonstrate that, under constrained training conditions, our multimodal approach significantly outperforms unimodal models. By providing a transparent and reproducible benchmark system, this study establishes a reliable foundation for future research in multimodal emotion recognition within dialogue contexts.

This work addresses the significant performance degradation in speech emotion recognition (SER) caused by the loss of paralinguistic information during the quantization of discrete speech tokens. To mitigate this issue, the authors propose a multi-layer fusion strategy combined with explicit integration of paralinguistic features: representations from multiple layers of a fine-tuned WavLM-Large model are fused via an attention mechanism and further augmented with acoustic features extracted using openSMILE. This approach effectively recovers semantic and emotional cues compromised in discrete tokenization. Experimental results demonstrate that the proposed method consistently narrows the performance gap between discrete tokens and continuous representations across several state-of-the-art neural audio codecs—including SpeechTokenizer, DAC, and EnCodec—thereby confirming its effectiveness and generalizability.

This study investigates whether widely adopted emotion embedding similarity metrics—such as those based on emotion2vec—genuinely reflect affective expressiveness in speech synthesis evaluation. By constructing adversarial voice samples and conducting human subjective listening experiments, the work reveals for the first time that such metrics in zero-shot emotional speech assessment are highly susceptible to interference from linguistic content and speaker identity, leading to significant divergence from human judgments. The findings demonstrate that emotion embeddings achieving high classification accuracy are ill-suited for similarity-based evaluation, as they tend to reward acoustic mimicry rather than authentic emotional expression. This research issues a critical caution against prevailing automatic evaluation paradigms and points toward more perceptually grounded directions for future benchmarking of emotional speech generation systems.