From Classification to Recommendation: Empirical Analysis of Audio Embedding Models Application for Content-Based Music Recommendation

2026-08-07

AI Summary

This study addresses the unclear applicability of pretrained audio embeddings in generative music recommendation, despite their strong performance in classification tasks. The authors systematically evaluate six representative audio encoders within content-based, sequential, and Semantic-ID–based generative recommender systems, and investigate how residual quantization affects the preservation of recommendation-relevant information. Their comprehensive comparison reveals that audio–text alignment and music-domain–specific representations yield superior performance, that sequential modeling substantially narrows performance gaps among encoders, and that excessively large Semantic-ID capacities not only offer no benefit but also induce instability. These findings provide crucial empirical evidence and novel insights for the design and deployment of audio representations in recommender systems.

Abstract

Pretrained audio representation models learned from large-scale corpora have achieved strong performance in audio classification and understanding. However, most existing models are optimized for objectives such as masked prediction, contrastive learning, or audio-text alignment, which do not necessarily produce representation spaces well-suited to recommender systems. Unlike classification, music recommender systems must capture item relationships shaped by subjective and behavior-dependent listener preferences. Although pretrained audio embeddings have been explored in conventional recommender systems, their effectiveness in the rapidly emerging paradigm of generative recommender systems remains underexplored. To address this gap, we systematically evaluate six representative audio encoders across three types of music recommender systems: content-based, sequential, and Semantic-ID-based generative recommender systems. We further investigate how residual-quantization design, including codebook width, quantization depth, and retained Semantic-ID prefixes, affects the preservation of recommendation-relevant information. Experiments on two music recommendation datasets show that audio-text-aligned and music-domain representations are generally more effective when pretrained embedding geometry is used directly, whereas interaction-based sequential training substantially reduces performance differences among encoders. We also find that increasing Semantic-ID capacity does not consistently improve generative recommender systems and may introduce substantial instability. These findings provide practical guidance for selecting audio encoders and designing audio-derived Semantic IDs for modern music recommender systems.

Problem

• audio embedding
• music recommendation
• generative recommender systems
• content-based recommendation
• Semantic-ID

Innovation

• audio embedding
• music recommendation
• generative recommender systems
• Semantic-ID
• residual quantization