MAEB: Massive Audio Embedding Benchmark

2026-02-17

AI Summary

This work addresses the lack of a unified, large-scale evaluation benchmark for audio embedding models, which has hindered comprehensive assessment of their generalization across speech, music, environmental sounds, and multilingual cross-modal tasks. To this end, we introduce MAEB, a large-scale audio embedding benchmark comprising 30 core tasks (98 in the extended MAEB+ variant), spanning over 100 languages and diverse audio types, and integrated into the MTEB multimodal evaluation ecosystem. Systematic evaluation of more than 50 prominent models reveals that no single model dominates across all tasks—contrastive audio-text models and speech-specialized architectures exhibit complementary strengths, clustering tasks remain challenging, and audio encoder performance on MAEB strongly correlates with its transfer effectiveness in audio large language models. This study establishes the first unified, multilingual, and multimodal evaluation framework for audio embeddings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

We introduce the Massive Audio Embedding Benchmark (MAEB), a large-scale benchmark covering 30 tasks across speech, music, environmental sounds, and cross-modal audio-text reasoning in 100+ languages. We evaluate 50+ models and find that no single model dominates across all tasks: contrastive audio-text models excel at environmental sound classification (e.g., ESC50) but score near random on multilingual speech tasks (e.g., SIB-FLEURS), while speech-pretrained models show the opposite pattern. Clustering remains challenging for all models, with even the best-performing model achieving only modest results. We observe that models excelling on acoustic understanding often perform poorly on linguistic tasks, and vice versa. We also show that the performance of audio encoders on MAEB correlates highly with their performance when used in audio large language models. MAEB is derived from MAEB+, a collection of 98 tasks. MAEB is designed to maintain task diversity while reducing evaluation cost, and it integrates into the MTEB ecosystem for unified evaluation across text, image, and audio modalities. We release MAEB and all 98 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.

Problem

• audio embedding
• benchmark
• multimodal evaluation
• cross-lingual audio
• model generalization

Innovation

• audio embedding benchmark
• multimodal evaluation
• cross-lingual audio understanding
• model specialization
• MTEB integration