Can Foundation Models Hear What Made That Sound? A Tiered Benchmark of Audio-Language Models and Traditional Classifiers for Closed-Set Sound Source Identification

2026-08-03

AI Summary

This study addresses closed-set sound source identification by proposing a hierarchical evaluation framework that systematically compares the performance of 11 audio models across 11 coarse-grained categories and 23 fine-grained subcategories. The framework organizes models into four tiers based on their input–output modalities, enabling fair comparison among heterogeneous architectures—including audio-language models (e.g., Gemini-3.1-Pro-Preview, Kimi-Audio-7B-Instruct), fixed-vocabulary classifiers (e.g., YAMNet, PANNs), and zero-shot or audio-grounding models (e.g., CLAP, SSLAM). Experimental results show that Gemini-3.1-Pro-Preview achieves 85.6% category-level F1 (56.7% at fine granularity), while Kimi-Audio excels among similarly sized models; notably, CLAP and SSLAM remain competitive even without a predefined candidate list. The work further reveals performance disparities and error patterns across semantic granularities, offering practical guidance for model selection in real-world applications.

Abstract

We benchmark eleven audio classification methods: five task-aware closed-set LLMs (four Gemini models plus open-weight Kimi-Audio-7B-Instruct), four fixed-vocabulary taggers (YAMNet, PANNs, Whisper-AT, and SSLAM), a zero-shot audio-text model (CLAP), and an audio-grounded LLM (BAT). We evaluate them on a closed-set sound-source identification task over 2,242 clips spanning 23 fine-grained classes and 11 categories. Since these methods differ fundamentally in how they receive the task and how outputs are scored, we group them into four evaluation tiers rather than one leaderboard, reporting macro Precision, Recall, F1, and false-negative rate per tier. The best model, Gemini-3.1-Pro-Preview, reaches 85.6 percent category-level F1 and 56.7 percent fine-grained F1. Kimi-Audio is competitive for its size, reaching 67.5 percent category-level F1 and 32.9 percent fine-grained F1, but fails to answer 1.6 percent of samples. SSLAM and CLAP match or exceed the best closed-set model at the category level without seeing the candidate list, but fall behind at the fine-grained level. Analyzing the Gemini models' chain-of-thought across 8,968 responses, we find that response length does not predict accuracy, an apparent "holistic judgment beats detailed analysis" effect is better explained as a difficulty confound, and wrong answers are stated confidently 92 to 100 percent of the time. We report full per-class confusion matrices and metrics for all eleven methods, identify the structural error modes behind most of the accuracy loss between granularities, and give practical guidance for choosing among these method families.

Problem

• sound source identification
• audio-language models
• closed-set classification
• fine-grained audio recognition
• model benchmarking

Innovation

• tiered benchmark
• audio-language models
• closed-set sound source identification
• chain-of-thought analysis
• foundation models