HASRD: Hierarchical Acoustic and Semantic Representation Disentanglement

2025-06-01

AI Summary

Semantic relevance and acoustic fidelity in speech representations are often mutually exclusive. This paper proposes a hierarchical disentangled self-supervised speech coding framework: a top-level codebook models discrete semantic tokens, while subsequent codebooks progressively encode acoustic residuals, thereby explicitly decoupling semantic and acoustic features. The method integrates vector quantization, multi-level codebook modeling, and an efficient encoder architecture, achieving both high ASR accuracy and significantly improved speech reconstruction quality. Experiments demonstrate that, compared to SpeechTokenizer, the approach reduces word error rate (WER) by 44% relatively, substantially improves mean opinion score (MOS) for speech reconstruction, and cuts bit rate by 50% (achieving 2× compression). To our knowledge, this is the first unified framework that simultaneously attains high recognition robustness and high-fidelity speech reconstruction.

Technology Category

Natural Language Processing: SpeechMachine Learning: Unsupervised & Self-Supervised LearningCognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web searchSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsGraph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Effective speech representations for spoken language models must balance semantic relevance with acoustic fidelity for high-quality reconstruction. However, existing approaches struggle to achieve both simultaneously. To address this, we introduce Hierarchical Acoustic and Semantic Representation Disentanglement (HASRD, pronounced `hazard'), a framework that factorizes self-supervised learning representations into discrete semantic and acoustic tokens. HASRD assigns the semantic representation to the first codebook, while encoding acoustic residuals in subsequent codebooks. This preserves ASR performance while achieving high-quality reconstruction. Additionally, we enhance HASRD's encoder efficiency, improving ASR performance without compromising reconstruction quality. Compared to SpeechTokenizer, HASRD achieves a 44% relative WER improvement, superior reconstruction quality, and 2x lower bitrate, demonstrating its effectiveness in disentangling acoustic and semantic information.

Problem

• Balancing semantic relevance and acoustic fidelity in speech representations
• Disentangling self-supervised learning into semantic and acoustic tokens
• Improving ASR performance without sacrificing reconstruction quality

Innovation

• Hierarchical disentanglement of acoustic and semantic representations
• Discrete semantic and acoustic tokens factorization
• Enhanced encoder efficiency without quality loss