Improving Inference-Time Optimisation for Vocal Effects Style Transfer with a Gaussian Prior

2025-05-16

AI Summary

This paper addresses the issue of unrealistic and highly biased outputs in inference-time optimization (ITO) for audio effect style transfer. To mitigate this, we propose a Bayesian calibration framework that introduces, for the first time, a learnable Gaussian prior into the ST-ITO framework, enabling maximum a posteriori (MAP) estimation in parameter space; the prior is learned from the DiffVox dataset. The method significantly improves fidelity and robustness of style matching under few-shot conditions: on the MedleyDB benchmark, parameter MSE decreases by 33%, and all style-matching metrics surpass those of blind estimation, nearest-neighbor baselines, and the original ST-ITO. A subjective evaluation involving 16 participants further confirms the superiority of the proposed approach in perceptual quality and stylistic accuracy.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Style Transfer with Inference-Time Optimisation (ST-ITO) is a recent approach for transferring the applied effects of a reference audio to a raw audio track. It optimises the effect parameters to minimise the distance between the style embeddings of the processed audio and the reference. However, this method treats all possible configurations equally and relies solely on the embedding space, which can lead to unrealistic or biased results. We address this pitfall by introducing a Gaussian prior derived from a vocal preset dataset, DiffVox, over the parameter space. The resulting optimisation is equivalent to maximum-a-posteriori estimation. Evaluations on vocal effects transfer on the MedleyDB dataset show significant improvements across metrics compared to baselines, including a blind audio effects estimator, nearest-neighbour approaches, and uncalibrated ST-ITO. The proposed calibration reduces parameter mean squared error by up to 33% and matches the reference style better. Subjective evaluations with 16 participants confirm our method's superiority, especially in limited data regimes. This work demonstrates how incorporating prior knowledge in inference time enhances audio effects transfer, paving the way for more effective and realistic audio processing systems.

Problem

• Enhancing vocal effects style transfer realism with Gaussian prior
• Reducing unrealistic biases in audio effects parameter optimization
• Improving inference-time optimization using preset-derived parameter constraints

Innovation

• Uses Gaussian prior for parameter optimization
• Applies maximum-a-posteriori estimation technique
• Improves vocal effects transfer accuracy