Affectron: Emotional Speech Synthesis with Affective and Contextually Aligned Nonverbal Vocalizations

2026-03-15

AI Summary

This work addresses the challenge of synthesizing diverse and emotionally consistent non-linguistic vocalizations—such as laughter and sighs—in open-domain scenarios, where data scarcity and the absence of explicit supervision hinder performance. To overcome these limitations, the authors propose Affectron, a novel framework that leverages a small-scale disentangled corpus and introduces a non-linguistic vocalization augmentation strategy to enrich both the variety and contextual distribution of such sounds. Furthermore, Affectron integrates a structured masking mechanism into a pre-trained text-to-speech model, enabling context-aware emotional expression without requiring large-scale annotated data. The method achieves state-of-the-art results in naturalness, diversity, and emotional consistency of synthesized non-linguistic vocalizations while preserving the fluency of the overall speech stream.

Technology Category

Natural Language Processing: SpeechCognitive Modeling & Cognitive Systems: Affective ComputingMachine Learning: Large Multimodal Models (LMMs)

Application Category

Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web searchEconomics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labelingUser Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

Nonverbal vocalizations (NVs), such as laughter and sighs, are central to the expression of affective cues in emotional speech synthesis. However, learning diverse and contextually aligned NVs remains challenging in open settings due to limited NV data and the lack of explicit supervision. Motivated by this challenge, we propose Affectron as a framework for affective and contextually aligned NV generation. Built on a small-scale open and decoupled corpus, Affectron introduces an NV-augmented training strategy that expands the distribution of NV types and insertion locations. We further incorporate NV structural masking into a speech backbone pre-trained on purely verbal speech to enable diverse and natural NV synthesis. Experimental results demonstrate that Affectron produces more expressive and diverse NVs than baseline systems while preserving the naturalness of the verbal speech stream.

Problem

• emotional speech synthesis
• nonverbal vocalizations
• contextual alignment
• affective cues
• data scarcity

Innovation

• nonverbal vocalizations
• emotional speech synthesis
• contextual alignment
• data augmentation
• structural masking