From A to B to A: Palindromic Zero-Shot Voice Conversion with Non-Parallel Data

2026-06-07

AI Summary

This work proposes a zero-shot cross-lingual voice conversion method that operates without parallel speech data. To address the challenges posed by the absence of explicit alignments and multilingual training corpora, the approach leverages WavLM speech representations and constructs synthetic training pairs from non-parallel source and target utterances via k-nearest neighbor retrieval, employing a “synthetic-to-real” supervised learning paradigm. Additionally, it incorporates a speaker loss derived from a pretrained speaker verification model to enhance target speaker consistency. Trained exclusively on English data, the method achieves high naturalness and strong speaker similarity across multiple languages, significantly outperforming existing baselines and demonstrating robust cross-lingual transfer capability and practical applicability.

Abstract

We present a voice conversion (VC) framework that utilizes K-Nearest Neighbors (KNN) retrieval over WavLM representations to align non-parallel source and target speech, constructing synthetic training pairs for supervised learning. The retrieved segments serve as synthetic inputs, while real target audio provides ground-truth outputs, forming a synthetic-to-real training paradigm that naturally supports multilingual data without requiring parallel corpora or explicit alignment. To ensure consistent target-speaker identity, we incorporate a speaker loss derived from a pretrained speaker verification model. Experiments across multiple languages demonstrate that the proposed approach achieves high naturalness and strong speaker similarity, outperforming competitive VC baselines, despite being trained exclusively on English data. Samples can be accessed at: https://palindromic-vc.github.io.

Problem

• zero-shot voice conversion
• non-parallel data
• speaker identity
• multilingual voice conversion
• voice conversion

Innovation

• Zero-Shot Voice Conversion
• Non-Parallel Data
• KNN Retrieval
• WavLM Representations
• Synthetic-to-Real Training