CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding

2026-06-02

AI Summary

Existing neural audio codecs often suffer from an imbalance between reconstruction quality and token efficiency, frequently encoding perceptually irrelevant background noise at the expense of critical speech content. This work proposes CleanCodec, the first audio codec framework to integrate a perception-guided denoising mechanism into its design, formulating audio tokenization as a selective information bottleneck problem that retains only perceptually salient features. By combining discrete tokenization with a perception-driven architecture, CleanCodec achieves state-of-the-art efficiency at an ultra-low bitrate of 12.5 tokens per second, significantly improving speaker similarity and speech intelligibility. Furthermore, it accelerates inference in downstream text-to-speech and voice conversion tasks by up to 17× compared to existing approaches.

Abstract

Neural audio codecs are a key component of speech processing pipelines, compressing audio into discrete tokens for downstream modeling. However, existing codecs struggle to balance reconstruction quality with token efficiency, often encoding perceptually irrelevant information such as background noise and recording artifacts at the expense of linguistically and acoustically meaningful content. We reframe audio tokenization as a selective information bottleneck problem and propose CleanCodec, a denoising audio codec which learns to encode only perceptually important features and discard imperceptible information. At just 12.5 tokens per second, CleanCodec achieves state-of-the-art tokenization efficiency, substantially outperforming existing codecs in speaker similarity and speech intelligibility. Evaluations on downstream text-to-speech and voice conversion tasks further demonstrate improved performance and up to 17x faster inference, highlighting significant efficiency gains.

Problem

• neural audio codecs
• token efficiency
• perceptual irrelevance
• speech tokenization
• information bottleneck

Innovation

• perceptually guided encoding
• denoising audio codec
• selective information bottleneck
• tokenization efficiency
• speech intelligibility