Melody-Lyrics Matching with Contrastive Alignment Loss

2025-07-31

AI Summary

This paper addresses cross-modal melody-to-lyrics matching (MLM), a task of retrieving singable lyrics given a symbolic melody. We propose a contrastive alignment learning framework that requires no aligned melody–lyrics annotations. Methodologically, we introduce syllable-level *sylphone* encoding to explicitly model phonemic, vocalic, and stress features; integrate self-supervised representation learning with a contrastive loss to achieve semantic alignment between melodies and lyrics in a shared latent space. Unlike prior generative approaches, we are the first to formally formulate MLM as a retrieval problem and learn cross-modal associations without explicit temporal alignment supervision. Experiments demonstrate significant improvements in lyrical singability and semantic matching accuracy. Open-sourced code and visualization examples further validate the method’s effectiveness and practicality in real-world music applications.

Technology Category

Machine Learning: Multimodal LearningNatural Language Processing: Language Grounding & Multi-modal NLPComputer Vision: Multi-modal Vision

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSearch and Retrieval-Augmented AI: Multilingual and cross-lingual Web searchWeb Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

The connection between music and lyrics is far beyond semantic bonds. Conceptual pairs in the two modalities such as rhythm and rhyme, note duration and syllabic stress, and structure correspondence, raise a compelling yet seldom-explored direction in the field of music information retrieval. In this paper, we present melody-lyrics matching (MLM), a new task which retrieves potential lyrics for a given symbolic melody from text sources. Rather than generating lyrics from scratch, MLM essentially exploits the relationships between melody and lyrics. We propose a self-supervised representation learning framework with contrastive alignment loss for melody and lyrics. This has the potential to leverage the abundance of existing songs with paired melody and lyrics. No alignment annotations are required. Additionally, we introduce sylphone, a novel representation for lyrics at syllable-level activated by phoneme identity and vowel stress. We demonstrate that our method can match melody with coherent and singable lyrics with empirical results and intuitive examples. We open source code and provide matching examples on the companion webpage: https://github.com/changhongw/mlm.

Problem

• Retrieve potential lyrics for given symbolic melodies
• Exploit relationships between melody and lyrics without alignment annotations
• Introduce syllable-level lyric representation for better matching

Innovation

• Self-supervised learning with contrastive alignment loss
• Syllable-level lyrics representation using sylphone
• Melody-lyrics matching without alignment annotations