Learning Robust Spatial Representations from Binaural Audio through Feature Distillation

2025-08-28

AI Summary

This work addresses the challenge of learning robust spatial representations from unlabeled binaural audio. We propose a self-supervised pre-training framework based on feature distillation, wherein clean binaural speech serves as the teacher signal to generate spatially consistent pseudo-labels; the student network is then trained via self-supervised distillation using augmented (noisy/reverberant) binaural speech—without any manual annotation. Our key contribution is an end-to-end, label-free spatial feature distillation mechanism that effectively decouples acoustic distortions from intrinsic spatial cues. In downstream direction-of-arrival (DOA) estimation, fine-tuned models significantly outperform fully supervised baselines and conventional signal processing methods under severe noise and high reverberation conditions, demonstrating the spatial robustness and generalization capability of the learned representations.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Natural Language Processing: Speech
• Intelligent Robots: State Estimation

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recently, deep representation learning has shown strong performance in multiple audio tasks. However, its use for learning spatial representations from multichannel audio is underexplored. We investigate the use of a pretraining stage based on feature distillation to learn a robust spatial representation of binaural speech without the need for data labels. In this framework, spatial features are computed from clean binaural speech samples to form prediction labels. These clean features are then predicted from corresponding augmented speech using a neural network. After pretraining, we throw away the spatial feature predictor and use the learned encoder weights to initialize a DoA estimation model which we fine-tune for DoA estimation. Our experiments demonstrate that the pretrained models show improved performance in noisy and reverberant environments after fine-tuning for direction-of-arrival estimation, when compared to fully supervised models and classic signal processing methods.

Problem

• Learning spatial representations from binaural audio without labels
• Improving direction-of-arrival estimation in noisy environments
• Using feature distillation to create robust audio spatial features

Innovation

• Feature distillation for binaural spatial learning
• Self-supervised pretraining without data labels
• Encoder transfer for robust DoA estimation