UniCoM: A Universal Code-Switching Speech Generator

2025-08-21

AI Summary

Code-switching (CS) is prevalent in real-world multilingual conversations, yet current speech technologies are hindered by the scarcity of high-quality CS data. To address this, we propose SWORDS—a novel cross-lingual lexical substitution algorithm that enforces part-of-speech constraints and semantic equivalence to ensure grammatical correctness, semantic fidelity, and naturalness in CS speech synthesis. Leveraging SWORDS, we develop a general-purpose code-mixing pipeline and construct CS-FLEURS, a large-scale, multilingual speech dataset covering diverse language pairs. Experimental evaluations demonstrate that CS-FLEURS achieves high intelligibility and naturalness in automatic speech recognition (ASR) and speech translation tasks—matching the performance of leading monolingual benchmarks. CS-FLEURS thus establishes the first large-scale, controllable, and high-fidelity CS benchmark for multilingual speech modeling.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Code-switching (CS), the alternation between two or more languages within a single speaker's utterances, is common in real-world conversations and poses significant challenges for multilingual speech technology. However, systems capable of handling this phenomenon remain underexplored, primarily due to the scarcity of suitable datasets. To resolve this issue, we propose Universal Code-Mixer (UniCoM), a novel pipeline for generating high-quality, natural CS samples without altering sentence semantics. Our approach utilizes an algorithm we call Substituting WORDs with Synonyms (SWORDS), which generates CS speech by replacing selected words with their translations while considering their parts of speech. Using UniCoM, we construct Code-Switching FLEURS (CS-FLEURS), a multilingual CS corpus designed for automatic speech recognition (ASR) and speech-to-text translation (S2TT). Experimental results show that CS-FLEURS achieves high intelligibility and naturalness, performing comparably to existing datasets on both objective and subjective metrics. We expect our approach to advance CS speech technology and enable more inclusive multilingual systems.

Problem

• Generating natural code-switching speech without altering semantics
• Addressing scarcity of datasets for multilingual code-switching technology
• Creating high-quality code-switching corpus for ASR and translation

Innovation

• Substituting words with synonyms algorithm
• Generates code-switching speech without altering semantics
• Multilingual corpus for speech recognition and translation