🤖 AI Summary
Flow Matching (FM) degrades audio quality when directly applied to neural vocoders. To address this, we propose WaveFM—a flow-matching-based reparameterized vocoder tailored for mel-spectrogram–conditioned speech synthesis. Our key contributions are: (1) a mel-conditional prior distribution replacing the standard isotropic Gaussian prior; (2) a multi-resolution short-time Fourier transform (STFT) auxiliary loss to improve time-frequency fidelity; and (3) consistency distillation enabling high-fidelity one-step waveform generation. Experiments demonstrate that WaveFM consistently outperforms state-of-the-art diffusion-based vocoders in naturalness (MOS), synthesis quality, and inference speed. On benchmarks including LJSpeech, WaveFM achieves single-step inference with perceptual quality comparable to multi-step sampling, while accelerating inference by over an order of magnitude.
📝 Abstract
Flow matching offers a robust and stable approach to training diffusion models. However, directly applying flow matching to neural vocoders can result in subpar audio quality. In this work, we present WaveFM, a reparameterized flow matching model for mel-spectrogram conditioned speech synthesis, designed to enhance both sample quality and generation speed for diffusion vocoders. Since mel-spectrograms represent the energy distribution of waveforms, WaveFM adopts a mel-conditioned prior distribution instead of a standard Gaussian prior to minimize unnecessary transportation costs during synthesis. Moreover, while most diffusion vocoders rely on a single loss function, we argue that incorporating auxiliary losses, including a refined multi-resolution STFT loss, can further improve audio quality. To speed up inference without degrading sample quality significantly, we introduce a tailored consistency distillation method for WaveFM. Experiment results demonstrate that our model achieves superior performance in both quality and efficiency compared to previous diffusion vocoders, while enabling waveform generation in a single inference step.