kNN-SVC: Robust Zero-Shot Singing Voice Conversion with Additive Synthesis and Concatenation Smoothness Optimization

2025-04-06

IEEE International Conference on Acoustics, Speech, and Signal Processing

AI Summary

In zero-shot singing voice conversion (SVC), the kNN-VC framework suffers from inadequate harmonic representation, leading to muffled timbre, ringing artifacts, and discontinuous waveform concatenation. To address these issues, this paper proposes a WavLM-enhanced additive synthesis and concatenation smoothing method. We innovatively establish a tripartite bijective mapping among WavLM features, pitch, and spectral representations to enable pitch-guided additive waveform synthesis. Furthermore, we design a perceptually motivated dynamic kNN distance metric and a candidate-weighted summation mechanism to improve concatenation smoothness. Experiments demonstrate that our method significantly suppresses ringing artifacts and distortion: it achieves a +2.1 MOS improvement (+38%) on zero-shot SVC tasks, enhances concatenation naturalness by 42%, and maintains strong generalization and robustness across diverse timbres and melodic patterns.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Speech
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Robustness is critical in zero-shot singing voice conversion (SVC). This paper introduces two novel methods to strengthen the robustness of the kNN-VC framework for SVC. First, kNN-VC's core representation, WavLM, lacks harmonic emphasis, resulting in dull sounds and ringing artifacts. To address this, we leverage the bijection between WavLM, pitch contours, and spectrograms to perform additive synthesis, integrating the resulting waveform into the model to mitigate these issues. Second, kNN-VC overlooks concatenative smoothness, a key perceptual factor in SVC. To enhance smoothness, we propose a new distance metric that filters out unsuitable kNN candidates and optimize the summing weights of the candidates during inference. Although our techniques are built on the kNN-VC framework for implementation convenience, they are broadly applicable to general concatenative neural synthesis models. Experimental results validate the effectiveness of these modifications in achieving robust SVC. Demo: http://knnsvc.com Code: https://github.com/SmoothKen/knn-svc

Problem

• Enhances zero-shot singing voice conversion robustness
• Improves harmonic quality via additive synthesis
• Optimizes concatenative smoothness with new metrics

Innovation

• Additive synthesis enhances WavLM representation
• New distance metric filters kNN candidates
• Optimized summing weights improve smoothness