MeanVoiceFlow2: Joint Optimization of Mean Flow and Content Encoder for Fast One-Step Zero-Shot Voice Conversion

2026-09-30

AI Summary

This study addresses the inference bottleneck in one-shot voice conversion models caused by their reliance on computationally intensive content encoders by proposing an end-to-end jointly optimized framework. Methodologically, the flow matching module and a lightweight content encoder are co-trained via knowledge distillation and reconstruction strategies. Furthermore, mean flow matching, diffusion GANs, sample mixing, and teacher-guided conditioning augmentation are integrated to enable rapid zero-shot voice conversion. Experimental results demonstrate that the proposed approach significantly improves perceptual quality while preserving speaker similarity, achieving an approximately ninefold increase in inference speed compared to the baseline.

Abstract

Flow-matching approaches to voice conversion (VC) have gained attention owing to their high speech quality and strong speaker similarity. Among them, one-step models such as MeanVoiceFlow are particularly attractive because they enable efficient inference; however, their reliance on a computationally intensive content encoder remains a bottleneck. We therefore propose MeanVoiceFlow2, a framework that jointly optimizes a flow-based conversion module and a computationally efficient content encoder. The model is trained through conversion distillation using MeanVoiceFlow and the reconstruction of real data. We further incorporate diffusion-GAN training with sample mixing and teacher-guided conditioning augmentation to enhance realism and disentanglement. Experiments on zero-shot VC showed that MeanVoiceFlow2 achieved higher perceptual quality and approximately $9\times$ faster inference than MeanVoiceFlow while maintaining comparable speaker similarity. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/meanvoiceflow2/.

Problem

• Zero-Shot Voice Conversion
• Flow Matching
• Content Encoder
• One-Step Inference
• Computational Bottleneck

Innovation

• Zero-Shot Voice Conversion
• Mean Flow
• Conversion Distillation
• Diffusion-GAN
• Content Encoder