HuLA: Prosody-Aware Anti-Spoofing with Multi-Task Learning for Expressive and Emotional Synthetic Speech

2025-09-25

AI Summary

Current anti-spoofing systems exhibit limited capability in detecting expressive and affective synthetic speech, primarily due to inadequate modeling of prosodic characteristics. To address this, we propose a prosody-aware two-stage multitask learning framework: in Stage I, a self-supervised backbone network jointly learns F0 prediction and voicing classification to explicitly model the dynamic prosody of natural speech; in Stage II, the learned prosodic representations are integrated into the spoof detection task for end-to-end co-optimization. This work is the first to deeply integrate explicit prosodic supervision with self-supervised learning, significantly enhancing robustness across domains, languages, and expressive synthetic speech. Our method substantially outperforms strong baselines on multiple challenging out-of-domain test sets—particularly excelling in high-expressivity scenarios, where generalization performance is markedly improved.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Multimodal Learning
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Current anti-spoofing systems remain vulnerable to expressive and emotional synthetic speech, since they rarely leverage prosody as a discriminative cue. Prosody is central to human expressiveness and emotion, and humans instinctively use prosodic cues such as F0 patterns and voiced/unvoiced structure to distinguish natural from synthetic speech. In this paper, we propose HuLA, a two-stage prosody-aware multi-task learning framework for spoof detection. In Stage 1, a self-supervised learning (SSL) backbone is trained on real speech with auxiliary tasks of F0 prediction and voiced/unvoiced classification, enhancing its ability to capture natural prosodic variation similar to human perceptual learning. In Stage 2, the model is jointly optimized for spoof detection and prosody tasks on both real and synthetic data, leveraging prosodic awareness to detect mismatches between natural and expressive synthetic speech. Experiments show that HuLA consistently outperforms strong baselines on challenging out-of-domain dataset, including expressive, emotional, and cross-lingual attacks. These results demonstrate that explicit prosodic supervision, combined with SSL embeddings, substantially improves robustness against advanced synthetic speech attacks.

Problem

• Detecting spoofed expressive and emotional synthetic speech attacks
• Leveraging prosodic cues to distinguish natural from synthetic speech
• Improving robustness against cross-lingual and emotional synthetic speech

Innovation

• Two-stage prosody-aware multi-task learning framework
• Self-supervised backbone with F0 and voiced/unvoiced tasks
• Joint optimization for spoof detection and prosody tasks