Causal Speech Enhancement with Predicting Semantics based on Quantized Self-supervised Learning Features

2024-12-26

AI Summary

To address the performance limitations of causal speech enhancement in real-time voice communication, this paper proposes the first end-to-end method integrating causal modeling with self-supervised learning (SSL)-based semantic prediction. Specifically, it estimates time-frequency masks solely from historical context while simultaneously predicting future quantized speech semantic tokens—e.g., phoneme-level SSL tokens. The approach innovatively combines causal convolution, feature-wise linear modulation (FiLM), vector quantization (VQ), and wav2vec 2.0–derived SSL features within a multi-task learning framework to explicitly model phonemic continuity in speech. Evaluated on the VoiceBank+DEMAND dataset, the method achieves a PESQ score of 2.88—significantly outperforming purely causal enhancement baselines. This demonstrates that incorporating semantic prediction substantially improves speech intelligibility and clarity in low-latency scenarios.

Technology Category

• Machine Learning: Causal Learning
• Natural Language Processing: Speech
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Real-time speech enhancement (SE) is essential to online speech communication. Causal SE models use only the previous context while predicting future information, such as phoneme continuation, may help performing causal SE. The phonetic information is often represented by quantizing latent features of self-supervised learning (SSL) models. This work is the first to incorporate SSL features with causality into an SE model. The causal SSL features are encoded and combined with spectrogram features using feature-wise linear modulation to estimate a mask for enhancing the noisy input speech. Simultaneously, we quantize the causal SSL features using vector quantization to represent phonetic characteristics as semantic tokens. The model not only encodes SSL features but also predicts the future semantic tokens in multi-task learning (MTL). The experimental results using VoiceBank + DEMAND dataset show that our proposed method achieves 2.88 in PESQ, especially with semantic prediction MTL, in which we confirm that the semantic prediction played an important role in causal SE.

Problem

• Real-time Voice Quality Enhancement
• Speech Clarity
• Audibility Improvement

Innovation

• Digital-Causal Integration
• Semantic Markers Prediction
• Speech Quality Enhancement