Controllable joint noise reduction and hearing loss compensation using a differentiable auditory model

2025-07-12

AI Summary

In speech enhancement for hearing-impaired listeners, jointly optimizing noise reduction (NR) and hearing loss compensation (HLC) remains challenging due to their conflicting objectives and the absence of ground-truth clean-and-compensated target signals. Method: This paper proposes an end-to-end multi-task learning framework that embeds a differentiable auditory model into a deep neural network. The model takes both noisy speech and individual audiograms as joint inputs and directly predicts time-frequency domain outputs that are simultaneously denoised and hearing-compensated. Contribution/Results: To our knowledge, this is the first approach enabling joint optimization and inference-stage controllable trade-off between NR and HLC within a differentiable auditory domain. Experiments show that our method matches state-of-the-art single-task models in objective metrics (e.g., PESQ, STOI), while supporting dynamic balancing of NR and HLC contributions—significantly improving speech intelligibility and naturalness even without clean-and-compensated reference data.

Technology Category

• Natural Language Processing: Learning & Optimization for NLP
• Machine Learning: Multimodal Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Deep learning-based hearing loss compensation (HLC) seeks to enhance speech intelligibility and quality for hearing impaired listeners using neural networks. One major challenge of HLC is the lack of a ground-truth target. Recent works have used neural networks to emulate non-differentiable auditory peripheral models in closed-loop frameworks, but this approach lacks flexibility. Alternatively, differentiable auditory models allow direct optimization, yet previous studies focused on individual listener profiles, or joint noise reduction (NR) and HLC without balancing each task. This work formulates NR and HLC as a multi-task learning problem, training a system to simultaneously predict denoised and compensated signals from noisy speech and audiograms using a differentiable auditory model. Results show the system achieves similar objective metric performance to systems trained for each task separately, while being able to adjust the balance between NR and HLC during inference.

Problem

• Joint noise reduction and hearing loss compensation optimization
• Lack of ground-truth targets in hearing loss compensation
• Balancing multi-task learning for denoising and compensation

Innovation

• Differentiable auditory model for joint optimization
• Multi-task learning balances NR and HLC
• Adjustable NR-HLC balance during inference