On the Design of Diffusion-based Neural Speech Codecs

2025-04-11

AI Summary

This work addresses low-bitrate neural speech coding (NSC), proposing the first systematic diffusion model (DM)-based NSC paradigm. Methodologically, it introduces a taxonomy of DM-NSC designs along two axes—conditioning mechanisms and output domains—to fully characterize the design space, and accordingly develops several novel diffusion architectures. The approach integrates conditional diffusion modeling, end-to-end training, and multi-granularity reconstruction objectives. Evaluation employs both subjective Mean Opinion Score (MOS) and objective metrics (PESQ, STOI, VISQOL). Results demonstrate that the proposed models significantly outperform GAN-based baselines at 1.6–3.2 kbps, achieving average MOS gains of 0.3–0.5 points. This confirms that diffusion models offer both structural soundness and perceptual superiority in speech coding, thereby establishing the first comprehensive, principled investigation of diffusion-based NSC.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Speech

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Content-based information diffusion

Abstract

Recently, neural speech codecs (NSCs) trained as generative models have shown superior performance compared to conventional codecs at low bitrates. Although most state-of-the-art NSCs are trained as Generative Adversarial Networks (GANs), Diffusion Models (DMs), a recent class of generative models, represent a promising alternative due to their superior performance in image generation relative to GANs. Consequently, DMs have been successfully applied for audio and speech coding among various other audio generation applications. However, the design of diffusion-based NSCs has not yet been explored in a systematic way. We address this by providing a comprehensive analysis of diffusion-based NSCs divided into three contributions. First, we propose a categorization based on the conditioning and output domains of the DM. This simple conceptual framework allows us to define a design space for diffusion-based NSCs and to assign a category to existing approaches in the literature. Second, we systematically investigate unexplored designs by creating and evaluating new diffusion-based NSCs within the conceptual framework. Finally, we compare the proposed models to existing GAN and DM baselines through objective metrics and subjective listening tests.

Problem

• Exploring systematic design of diffusion-based neural speech codecs
• Comparing diffusion models with GANs in speech coding performance
• Proposing categorization and evaluating new diffusion-based NSC designs

Innovation

• Diffusion Models for neural speech codecs
• Categorization based on conditioning domains
• Systematic evaluation of new NSC designs