🤖 AI Summary
This work addresses the challenge of efficiently and disentangledly representing semantic and acoustic information in neural audio codecs by proposing HybridCodec, a novel codec that integrates a dual-stream architecture with self-supervised learning (SSL) representation distillation. During training, HybridCodec achieves strong disentanglement between semantic and acoustic features through semantic distillation, while eliminating the need for SSL models during inference. By unifying semantic distillation with a dual-stream structure, the method simultaneously ensures semantic specificity and high-fidelity reconstruction, significantly enhancing inference efficiency. Experimental results demonstrate that HybridCodec achieves superior semantic disentanglement and competitive reconstruction quality on in-domain tasks, exhibits robustness in cross-domain and zero-shot cross-lingual scenarios, and attains a threefold speedup in inference compared to existing dual-stream models.
📝 Abstract
The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models. New codec architectures with semantic and acoustic disentanglement have emerged. There are two main approaches to introduce semantic information into codec models: one distills semantic information from SSL representations into the first RVQ layer, while the other maintains separate streams for semantic and acoustic features. We propose HybridCodec, a unified architecture that combines both paradigms. It employs separate semantic and acoustic branches while distilling SSL representations into the semantic stream. This design ensures strong disentanglement without requiring an SSL model during inference. HybridCodec shows superior semantic specialization (RVQ-1) on in-domain test set and competitive reconstruction (RVQ-all). We demonstrate its robustness in out-of-domain and zero-shot cross-lingual settings, achieving a 3x speedup over existing dual-stream models.