🤖 AI Summary
This work addresses the limitations of existing automatic singing quality assessment methods, which often rely on a single modality and struggle to jointly evaluate lyrical accuracy and musical expressiveness. To overcome this, the authors propose MusicJudge, a framework that leverages block-level multimodal alignment to simultaneously assess lyric correctness and pitch–rhythm fidelity. Its key innovations include integrating semantic embeddings, lexical similarity, and phoneme alignment to accurately identify semantically coherent lyric blocks, as well as introducing a modality-guided LoRA fine-tuning strategy to enhance the robustness of automatic speech recognition (ASR) for sung transcription. Experimental results demonstrate that MusicJudge achieves high agreement with human expert ratings across multiple datasets and significantly outperforms current state-of-the-art approaches, confirming its effectiveness and generalization capability.
📝 Abstract
Automatic singing quality assessment (SQA) requires evaluating lyrical correctness and musical fidelity while handling expressive variations. However, existing systems largely rely on either acoustic cues or lyric transcriptions exclusively, limiting holistic performance evaluation. Furthermore, their integration is non-trivial due to challenges in robust singing transcription amid melisma, vibrato, and tempo elasticity. To this end, we propose MusicJudge, a modality-guided framework for automated SQA that performs block-aligned multimodal analysis by coupling lyric correctness with pitch-rhythm fidelity. It detects semantically meaningful lyric blocks using multi-signal matching that integrates semantic embeddings, lexical similarity, and phonetic alignment. To improve singing audio transcription, we introduce Modality-Guided LoRA for ASR fine-tuning. Experiments across datasets demonstrate strong agreement with human expert judgments and validate the generalizability of MusicJudge.