MPEcho: A Melody and Phoneme-Aware Generative Framework for Controllable Cover Song Generation

2026-07-29

AI Summary

Existing singing voice conversion methods struggle to simultaneously preserve melodic fidelity and ensure lyrical accuracy, often resulting in high phoneme error rates. This work proposes Phonsa, a novel framework that introduces explicit phoneme-level conditioning into singing voice conversion for the first time, coupled with a Whisper-driven automatic phoneme transcription module to mitigate the scarcity of high-quality annotated singing phonemes. Built upon the SongEcho architecture, Phonsa integrates a phoneme encoder and a duration regulator, leveraging precise phoneme alignment information to enable accurate joint control over melody and lyrics. Experimental results demonstrate that the proposed approach significantly reduces phoneme error rates, enhances lyrical accuracy, and improves overall generation quality, thereby validating the superior alignment capability of Phonsa and the end-to-end generation performance of MPEcho.

Abstract

Cover song generation (CSG) should preserve the melodic and linguistic content of a reference song while recreating the remaining musical components. The state-of-the-art model SongEcho utilizes $F_0$ sequences and voiced/unvoiced (V/UV) tags for conditioning; however, implicit linguistic information from V/UV tags cannot guarantee lyric accuracy, leading to a high phoneme error rate (PER). Inspired by singing voice synthesis (SVS), we propose MPEcho, which integrates a phoneme encoder and a length regulator (LR) into the SongEcho framework. By providing explicit phoneme-level conditioning and precise temporal boundaries, MPEcho significantly reduces PER. To enable this, we developed Phonsa, a Whisper-based automatic transcription model that provides high-precision phoneme-level annotations for singing voices, overcoming the scarcity of high-quality audio-phoneme pairs. Experimental results validate the effectiveness of Phonsa for alignment and MPEcho for end-to-end CSG. The audio samples, code and weights can be accessed from https://lonian6.github.io/MPEcho.github.io/.

Problem

• cover song generation
• phoneme accuracy
• melodic preservation
• linguistic content
• phoneme error rate

Innovation

• phoneme-aware
• cover song generation
• length regulator
• singing voice synthesis
• automatic phoneme transcription