Pushing the Frontier of Full-Song Generation: Hierarchical Autoregressive Planning Meets Flow-Matching Rendering

2026-07-22

AI Summary

This work proposes a unified framework for full-song generation that simultaneously supports lyric-conditioned singing, instrumental music, and cover song synthesis. The approach integrates hierarchical autoregressive modeling with continuous flow-matching rendering and introduces a dual-level melody guidance mechanism to preserve the original melodic structure in cover songs. For the first time, reward-driven optimization strategies—including DPO, GRPO, and OPD—are applied to full-song generation. The system employs a semantic-aware 8-codebook RVQ audio tokenizer, a Hybrid-LM language model, and a FullDiT decoder. Evaluations on multilingual automatic metrics and the Artificial Analysis Music with Vocals leaderboard demonstrate state-of-the-art performance in song completeness, audio fidelity, and musicality.

Abstract

In this report, we present a unified song generation framework capable of producing high-quality full-length music from lyrics, text descriptions, and musical attributes. The proposed framework supports three tasks: Lyrics-to-Song Generation, which generates complete songs from text descriptions, lyrics, and musical attributes; Instrumental Music Generation, which creates music without vocals; and Cover Song Generation, which reinterprets existing songs with different styles while preserving their melodic content. Architecturally, our system consists of four main components: a semantic-aware tokenizer, hybird-LM, FullDiT, and a two-level melody module. The tokenizer encodes audio into 8-codebook RVQ tokens for efficient discrete music representation. Based on these tokens, hybird-LM performs hierarchical autoregressive audio-token modeling for full-song generation. To improve audio fidelity, FullDiT performs full-song flow matching in a continuous VAE latent space conditioned on codec tokens, lyrics, and text captions. For cover song generation, the melody module extracts and discretizes melody cues from reference audio to guide generation while preserving the original melodic content. Finally, we investigate DPO, GRPO, and OPD as reward-based post-training strategies for hybird-LM and apply flow-based GRPO to FullDiT to improve musicality and rendering quality. Experimental results on a multilingual automatic benchmark, complemented by the Artificial Analysis Music with Vocals leaderboard, show that the proposed framework achieves competitive performance in the evaluated settings.

Problem

• full-song generation
• lyrics-to-song
• cover song generation
• instrumental music generation
• music rendering

Innovation

• hierarchical autoregressive modeling
• flow-matching rendering
• semantic-aware RVQ tokenizer
• melody-preserving cover generation
• reward-based post-training