Semantic-Aware Interpretable Multimodal Music Auto-Tagging

2025-05-22

AI Summary

Automatic music tagging suffers from insufficient model interpretability, undermining trustworthiness and user controllability. To address this, we propose a semantic-aware multimodal interpretable tagging framework that innovatively integrates signal processing, deep representation learning, music ontology modeling, and natural language processing. We introduce, for the first time, a semantic-clustering–based multimodal feature grouping and weighting mechanism, coupled with an EM algorithm for dynamic optimization of group weights. This design ensures state-of-the-art (SOTA) tagging accuracy while rendering the decision process fully transparent. The framework supports traceable semantic attribution paths, significantly enhancing model trustworthiness and human-AI collaboration capability. Extensive evaluations on mainstream benchmarks validate both its effectiveness and practical utility.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Syntax — Tagging, Chunking & Parsing
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Music auto-tagging is essential for organizing and discovering music in extensive digital libraries. While foundation models achieve exceptional performance in this domain, their outputs often lack interpretability, limiting trust and usability for researchers and end-users alike. In this work, we present an interpretable framework for music auto-tagging that leverages groups of musically meaningful multimodal features, derived from signal processing, deep learning, ontology engineering, and natural language processing. To enhance interpretability, we cluster features semantically and employ an expectation maximization algorithm, assigning distinct weights to each group based on its contribution to the tagging process. Our method achieves competitive tagging performance while offering a deeper understanding of the decision-making process, paving the way for more transparent and user-centric music tagging systems.

Problem

• Enhancing interpretability in music auto-tagging systems
• Leveraging multimodal features for meaningful music tagging
• Balancing performance and transparency in tagging decisions

Innovation

• Leverages multimodal features from diverse technologies
• Uses semantic clustering for enhanced interpretability
• Applies expectation maximization for weighted feature contribution