USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding

2026-06-04

AI Summary

This work addresses the limitations of existing audio encoders in multi-domain generalization and alignment with large audio language models, which hinder their capacity for universal audio understanding. The authors propose a universal audio encoder that integrates knowledge from both self-supervised and supervised foundation models. To mitigate teacher model mismatch, they introduce domain-aware distillation and, for the first time, extend the approach to the music domain. Furthermore, a two-stage supervised distillation strategy is employed to enhance downstream task performance. The resulting model scales to a billion parameters and achieves state-of-the-art or competitive results across probing tasks and evaluations based on large audio language models.

Abstract

Audio encoders are critical to modern audio applications as large language models (LLMs) increasingly rely on a single encoder for diverse inputs. While self-supervised learning (SSL) has yielded strong domain-specific encoders like speech or music experts, multi-domain approaches like USAD and SPEAR remain limited in coverage and evaluation. Recent studies also suggest supervised encoders align better with audio LLMs. We present USAD 2.0, a universal encoder integrating knowledge from both SSL and supervised foundation models. USAD 2.0 introduces domain-aware distillation to address teacher mismatch, extends coverage to the music domain, and adds second-stage supervised distillation for downstream use. We further scale the model to one billion parameters via depth scaling. Experiments show USAD 2.0 achieves strong or state-of-the-art performance across probing and LLM-based evaluations.

Problem

• universal audio understanding
• audio encoders
• multi-domain
• representation distillation
• foundation models

Innovation

• domain-aware distillation
• universal audio encoder
• representation distillation
• supervised foundation models
• scaling