Open-Source Full-Duplex Conversational Datasets for Natural and Interactive Speech Synthesis

2025-09-04

AI Summary

Current conversational TTS systems suffer from a scarcity of natural, interactive bilingual speech data, hindering effective modeling of authentic dialogue phenomena—such as overlapping speech, backchannel responses, and laughter. To address this, we introduce the first high-quality, bilingual (Chinese–English), full-duplex, spontaneous dialogue speech corpus (15 hours of multi-track recordings), covering everyday topics and genuine interactive behaviors. We propose an open-source dual-channel acquisition protocol and a fine-grained transcription annotation schema explicitly designed to capture overlapping utterances, nonverbal vocalizations, and feedback responses. Fine-tuning TTS models on this dataset yields statistically significant improvements over strong baselines in both objective metrics and subjective evaluations—particularly in speech naturalness and dialogue realism. This work establishes a foundational bilingual resource and methodological framework for advancing conversational speech synthesis.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Full-duplex, spontaneous conversational data are essential for enhancing the naturalness and interactivity of synthesized speech in conversational TTS systems. We present two open-source dual-track conversational speech datasets, one in Chinese and one in English, designed to enhance the naturalness of synthesized speech by providing more realistic conversational data. The two datasets contain a total of 15 hours of natural, spontaneous conversations recorded in isolated rooms, which produces separate high-quality audio tracks for each speaker. The conversations cover diverse daily topics and domains, capturing realistic interaction patterns including frequent overlaps, backchannel responses, laughter, and other non-verbal vocalizations. We introduce the data collection procedure, transcription and annotation methods. We demonstrate the utility of these corpora by fine-tuning a baseline TTS model with the proposed datasets. The fine-tuned TTS model achieves higher subjective and objective evaluation metrics compared to the baseline, indicating improved naturalness and conversational realism in synthetic speech. All data, annotations, and supporting code for fine-tuning and evaluation are made available to facilitate further research in conversational speech synthesis.

Problem

• Creating open-source conversational datasets for TTS
• Enhancing synthesized speech naturalness and interactivity
• Addressing lack of realistic dual-speaker interaction data

Innovation

• Open-source dual-track conversational speech datasets
• Data collection with isolated recording for separate tracks
• Fine-tuning TTS model for improved conversational realism