MiDashengLM-Spatial: Unifying General Audio Understanding and Spatial Awareness

2026-10-07

AI Summary

This work addresses the bottleneck of existing large audio models in simultaneously achieving general understanding and spatial perception by proposing the first open-source, unified end-to-end large audio model. Methodologically, it integrates the Spatial-Dasheng spatial encoder and introduces a novel hierarchical semantic-to-spatial conditioning injection module to enable synergistic dual capabilities, alongside constructing a large-scale spatial audio data synthesis pipeline. Experimental results demonstrate that the proposed model significantly outperforms existing methods on spatial perception benchmarks while maintaining state-of-the-art performance on monaural general tasks. This approach effectively bridges the gap between general audio understanding and spatial perception.

Abstract

Large audio-language models (LALMs) have achieved strong performance in general audio understanding, yet most are designed for monaural input and discard the inter-channel cues essential for spatial perception. In contrast, existing spatial audio-language models are purpose-built for spatial tasks and fail to capitalize on the general understanding capabilities of monaural LALMs. We present MiDashengLM-Spatial, the first open-source end-to-end unified audio-language model, to our knowledge, which supports both general audio understanding and spatial awareness within a single architecture. It extends MiDashengLM with a spatial audio encoder, Spatial-Dasheng, integrated through a hierarchical semantic-to-spatial conditioning module that injects intermediate semantic representations into the spatial branch at multiple depths while preserving the original semantic pathway. To provide spatial audio-language supervision at scale, we develop a data synthesis pipeline that renders diverse spatial acoustic scenes with scene-level spatial descriptions and question-answer pairs. Experiments show that Spatial-Dasheng achieves strong performance on sound event localization and detection in real-world scenes, and that MiDashengLM-Spatial substantially outperforms existing LALMs on spatial understanding and reasoning benchmarks. Meanwhile, it remains competitive with state-of-the-art 8B-scale LALMs on diverse monaural benchmarks, demonstrating that spatial awareness can be acquired without compromising general audio understanding. The source code and model checkpoint are available at https://github.com/xiaomi-research/midashenglm-spatial and https://huggingface.co/mispeech/midashenglm-spatial.

Problem

• Large Audio-Language Models
• Spatial Awareness
• General Audio Understanding
• Unified Architecture

Innovation

• unified audio-language model
• spatial audio encoder
• hierarchical conditioning module
• data synthesis pipeline
• spatial awareness