🤖 AI Summary
Diffusion-based text-to-speech synthesis suffers from slow reverse sampling, severe early-stage denoising errors causing distortion, and inefficient training. To address these issues, this paper proposes a content-transfer-guided hierarchical progressive denoising framework. Its core contributions are: (1) a novel skip-step parameter τ, enabling each network layer to directly model the inverse of τ-step forward noise addition—thereby drastically reducing inference steps; and (2) replacing conventional x₀/ε₀ regression with multi-step forward noise prediction as an intermediate supervision target, enhancing gradient consistency and representation robustness. Experiments demonstrate over 50% reduction in inference steps, superior speech fidelity compared to state-of-the-art vocoders, and strong generalization to unseen utterances—achieving an optimal trade-off between efficiency and high-fidelity generation.
📝 Abstract
Diffusion based vocoders have been criticised for being slow due to the many steps required during sampling. Moreover, the model's loss function that is popularly implemented is designed such that the target is the original input $x_0$ or error $epsilon_0$. For early time steps of the reverse process, this results in large prediction errors, which can lead to speech distortions and increase the learning time. We propose a setup where the targets are the different outputs of forward process time steps with a goal to reduce the magnitude of prediction errors and reduce the training time. We use the different layers of a neural network (NN) to perform denoising by training them to learn to generate representations similar to the noised outputs in the forward process of the diffusion. The NN layers learn to progressively denoise the input in the reverse process until finally the final layer estimates the clean speech. To avoid 1:1 mapping between layers of the neural network and the forward process steps, we define a skip parameter $ au>1$ such that an NN layer is trained to cumulatively remove the noise injected in the $ au$ steps in the forward process. This significantly reduces the number of data distribution recovery steps and, consequently, the time to generate speech. We show through extensive evaluation that the proposed technique generates high-fidelity speech in competitive time that outperforms current state-of-the-art tools. The proposed technique is also able to generalize well to unseen speech.