neural vocoder conditioning

Flow Matching (FM) degrades audio quality when directly applied to neural vocoders. To address this, we propose WaveFM—a flow-matching-based reparameterized vocoder tailored for mel-spectrogram–conditioned speech synthesis. Our key contributions are: (1) a mel-conditional prior distribution replacing the standard isotropic Gaussian prior; (2) a multi-resolution short-time Fourier transform (STFT) auxiliary loss to improve time-frequency fidelity; and (3) consistency distillation enabling high-fidelity one-step waveform generation. Experiments demonstrate that WaveFM consistently outperforms state-of-the-art diffusion-based vocoders in naturalness (MOS), synthesis quality, and inference speed. On benchmarks including LJSpeech, WaveFM achieves single-step inference with perceptual quality comparable to multi-step sampling, while accelerating inference by over an order of magnitude.

Traditional neural speech codecs struggle to disentangle linguistic content, speaker identity, and prosody, often resulting in poor prosody preservation during voice conversion. This work proposes a prosody-oriented codec that models prosody as a conditional residual guided by textual and speaker embeddings, while capturing prosodic variations unexplained by content or speaker through discrete bottleneck representations. By integrating low-frequency Mel-band modeling and training on same-speaker paired data, the method effectively enhances prosody retention. Experimental results demonstrate that the proposed approach significantly improves prosody transfer in voice conversion tasks and substantially reduces source speaker timbre leakage.

To address the challenges of subjective quality assessment and the inefficiency of purely data-driven models in speech/audio coding, this paper proposes a tightly integrated hybrid neural coding framework that synergistically combines model-driven and data-driven paradigms. Methodologically, it introduces a novel multi-level hybrid architecture that deeply couples psychoacoustic-weighted loss, customized time-frequency domain prediction (TF-Codec/MDCTNet), an LPCNet-based backbone, and a neural post-processing module, trained end-to-end via an autoencoder paradigm. The core contribution lies in systematically bridging the performance gap between classical signal modeling and end-to-end deep learning. Experimental results demonstrate that, at ultra-low bitrates of 1.6–3.2 kbps, the proposed method achieves a P.808 MOS gain of ≥0.5 over baselines, yielding subjective audio quality approaching that of wideband codecs, while increasing computational overhead by less than 15%.

Existing speech tokenization methods rely on audio compressors, incurring high computational overhead and poor cross-domain generalization. This paper proposes dMel—a training-free, streaming-capable, and robust discrete speech representation—achieved by intensity-based binning of energy per frequency band in log-Mel spectrograms, enabling lightweight tokenization. Its core innovation lies in the first unified optimization of text-to-speech (TTS) and automatic speech recognition (ASR) within a single LM-style Transformer architecture; it employs parallel high-dimensional token encoding/decoding to jointly balance efficiency and representational capacity. Experiments demonstrate that dMel matches or surpasses task-specific models in both synthesis and recognition performance, while significantly reducing computational complexity and deployment barriers. By eliminating the need for auxiliary neural compressors and dedicated training, dMel establishes a simple, efficient, and scalable representation paradigm for speech foundation models.

To address the low compression efficiency and poor fidelity of discrete quantization for high-dimensional audio signals, this paper introduces WavTokenizer—the first efficient discrete encoder-decoder specifically designed for audio language modeling. It innovatively constructs a wide vector-quantized (VQ) codebook space, integrated with an extended-context Transformer, multi-scale GAN discriminators, and an inverse short-time Fourier transform (iSTFT)-based reconstruction architecture, enabling compression of 1-second 24 kHz audio into only 40–75 semantically rich tokens. WavTokenizer achieves state-of-the-art performance across speech, music, and general audio reconstruction: it attains a new industry-leading UTMOS score (+0.32), improves VQ codebook utilization by 37%, and significantly enhances both perceptual quality and semantic consistency. The model is lightweight, open-source, and natively compatible with downstream audio generation tasks.

This work addresses the mismatch between the optimization objectives of existing neural audio codecs and the autoregressive prediction requirements of language models, which introduces acoustic uncertainty in discrete tokens and increases language model perplexity. Without altering the codec or large language model architectures, the authors propose a language-model-oriented joint optimization framework that enhances token predictability through a Medusa-style multi-step future token prediction head, aligns audio-text semantics via a memory-bank-based contrastive loss, and enables end-to-end training using a differentiable Gumbel bridge. Experiments demonstrate that the proposed method achieves a 61.6% accuracy (+12.1%) on the SALMon speech coherence task with a 35-point reduction in perplexity, and improves Mel-distance by 5.0% on Codec-SUPERB-tiny while preserving both reconstruction fidelity and token learnability.

This work addresses the ambiguity in labeling resynthesized audio generated by neural audio codecs—a class of models that combine compression and synthesis capabilities—within the context of voice spoofing detection. The study presents the first systematic analysis of this labeling challenge, introducing an extended version of the ASVspoof 5 dataset and proposing multiple annotation strategies tailored to resynthesized audio. A unified evaluation framework is designed to assess the impact of different labeling approaches on anti-spoofing systems, leveraging resynthesis techniques that integrate neural codecs with vocoders. Experimental results demonstrate that the choice of annotation strategy significantly influences detection performance, offering critical insights for future dataset construction and evaluation protocols in audio deepfake detection research.

Discrete audio representations in speech language models often degrade downstream task performance due to information loss. To address this, this work proposes a hybrid discrete-continuous modeling approach that jointly represents speech using temporally compressed discrete tokens and dimensionality-reduced continuous residuals. The method introduces a novel encoder-decoder architecture incorporating fusion-focused modulation and a hybrid Transformer design, enabling autoregressive inference in the discrete domain while simultaneously leveraging non-autoregressive prediction and continuous residual upsampling. This approach achieves the first effective integration of discrete and continuous representations, substantially reducing the number of autoregressive steps while preserving speaker characteristics and fine-grained acoustic details. Experimental results demonstrate clear performance gains over purely discrete baselines.

This work addresses the vulnerability of automatic speech recognition (ASR) systems to adversarial perturbations—distortions imperceptible to humans yet capable of inducing transcription errors. The authors propose a neural audio codec based on residual vector quantization (RVQ) that introduces a discrete bottleneck in the signal pathway to suppress adversarial noise while preserving linguistic content. Their analysis reveals a non-monotonic trade-off between quantization depth and robustness, demonstrating that intermediate RVQ depths optimally balance content fidelity and adversarial resilience. Notably, the study establishes, for the first time, a strong correlation between discrete codebook alterations and transcription errors. Experimental results show that the proposed method significantly reduces word error rates across multiple attack types, outperforming conventional compression-based defenses and maintaining robustness even under adaptive attacks.

This work proposes a training-free audio style transfer method that achieves high-quality, controllable audio morphing in the discrete latent space of neural audio codecs via residual vector quantization (RVQ). By combining hierarchical grouping of RVQ layers with sequence-level optimization, the approach leverages a user-provided timbral “palette” through a grouped token transfer strategy and a beam search matcher constrained by continuity priors. This enables precise transfer of target timbral details while preserving the rhythmic structure of the source audio. As the first training-free, real-time controllable audio morphing framework operating directly in the RVQ token domain, the method has been successfully deployed as a VST3/AU plugin supporting chunked rendering and an extensible timbre palette, balancing high fidelity with interactive efficiency.