LL-SDR: Low-Latency Speech enhancement through Discrete Representations

2026-03-10

arXiv.org

AI Summary

This work addresses the limitations of existing speech enhancement methods, which predominantly rely on continuous representations and struggle to efficiently disentangle speech from noise, while the efficacy of discrete-token approaches remains unclear. The authors propose a low-latency, non-autoregressive discrete speech enhancement framework featuring two key innovations: a Variance-Ordered Residual Vector Quantizer (VO-RVQ) that explicitly separates speech and noise distributions, and a latent-space discriminator that aligns enhanced embeddings with semantic embeddings. Requiring only 40G MACs, the method achieves real-time factors of 0.01 on GPU and 0.24 on CPU, outperforming continuous baselines and matching the quality of autoregressive models—demonstrating that high-quality speech enhancement can be realized with both high efficiency and minimal latency.

Abstract

Many speech enhancement (SE) methods rely on continuous representations. Recently, discrete audio tokens have been explored to enable autoregressive generation for SE. However, it remains unclear whether discretization itself consistently improves SE performance. In this paper, we introduce LL-SDR, a token-based speech enhancement framework that explicitly leverages discretization to better separate speech and noise. Our first contribution is a Variance-Ordered Residual Vector Quantizer (VO-RVQ), designed to disentangle speech and noise distributions during tokenization. Second, we propose a latent-space discriminator to better align enhanced embeddings with semantic embeddings. Experiments show that LL-SDR outperforms continuous baselines and matches the performance of autoregressive token-based approaches, while enabling lightweight, low-latency speech enhancement in both reverberant and non-reverberant noisy environments. Demos and source code are available at our project websites.

Problem

• speech enhancement
• discrete representations
• token-based
• noise separation
• latency

Innovation

• discrete representations
• speech enhancement
• vector quantization
• low-latency inference
• latent-space discriminator