UniVocal: Unified Speech-Singing Code-Switching Synthesis

2026-06-01

AI Summary

This work proposes UniVocal, a unified framework for seamless synthesis of speech and singing driven by textual semantics without requiring explicit labels. The model implicitly infers the vocal mode from textual context and incorporates a two-stage curriculum learning strategy, synthetic data augmentation, and a semantic-acoustic alignment approach for data generation. It further introduces a refined cent token representation and a Chain-of-Thought prosody planning mechanism. The study pioneers the code-switching synthesis task between speech and singing, establishes a multi-scenario evaluation benchmark named SCSBench, and achieves state-of-the-art performance on this benchmark while maintaining competitive results on conventional text-to-speech and singing voice synthesis tasks.

Abstract

We propose UniVocal, a unified framework that implicitly infers vocal modes from text context to pioneer Speech-Singing Code-Switching (SCS) Synthesis - a task where transitions are autonomously driven by textual semantics, akin to seamless human language blending. Unlike single-mode generation or systems relying on switching-control tags, our proposed UniVocal implicitly infers vocal modes solely from text context. To achieve this, we employ a data-efficient two-stage curriculum learning strategy that progressively trains a competitive TTS system to acquire the desired SCS capability. Addressing data scarcity, we introduce a scalable pipeline to synthesize diverse code-switching data that is both semantically and acoustically natural, alongside a new multi-scenario benchmark, SCSBench. To address limitations of semantic tokenizers in capturing acoustic details, we also introduce refined cent token and Chain-of-Thought (CoT) generation for planning prosody before content generation, effectively enhancing empathetic speech generation and singing melody. Experimental results demonstrate that UniVocal achieves state-of-the-art performance on SCSBench while maintaining competitive performance on regular speech and singing tasks. Audio samples are available at https://project-univocal-demo.github.io/demo/. The code and dataset are released at https://github.com/FunAudioLLM/FunResearch/tree/main/UniVocal.

Problem

• Speech-Singing Code-Switching
• text-to-speech
• singing synthesis
• vocal mode inference
• prosody planning

Innovation

• Speech-Singing Code-Switching
• implicit mode inference
• Chain-of-Thought prosody planning
• cent token
• curriculum learning