SSL-GMMVC: Interpretable Voice Conversion via Locally Linear GMM Transforms in Self-Supervised Representation Space

2026-06-08

AI Summary

This work addresses the limited interpretability and inadequate structural alignment in feature spaces commonly encountered in voice conversion by proposing a local linear transformation method grounded in self-supervised speech representations. The approach models source–target feature pairs using Gaussian mixture models and implements localized affine mappings through posterior-weighted transformations. Crucially, it explicitly links Gaussian component selection to phonetic structures, thereby endowing the conversion process with interpretable scaling and rotation properties. Experimental results demonstrate that the method significantly improves speaker similarity while preserving speech naturalness and intelligibility. Notably, under constrained covariance settings, its performance surpasses that of deep learning baselines as the number of mixture components increases.

Abstract

We introduce SSL-GMMVC, an interpretable voice conversion method in self-supervised speech space. The method models paired source-target features with a Gaussian mixture model and performs conversion as a posterior-weighted sum of affine transforms. This yields locally linear transformations that adapt to heterogeneous feature-space structure while remaining analytically tractable. Through objective and subjective evaluations, we show that SSL-GMMVC improves speaker similarity with comparable intelligibility and naturalness, and that even a constrained covariance variant surpasses a deep learning baseline as the number of mixture components increases. Further analyses link component selection to phonetic structure and reveal interpretable scaling and rotation in the learned transforms. These findings highlight SSL-GMMVC as an effective, analyzable framework for voice conversion.

Problem

• voice conversion
• self-supervised learning
• interpretability
• Gaussian mixture model
• speech representation

Innovation

• voice conversion
• self-supervised learning
• Gaussian mixture model
• interpretable AI
• locally linear transform