SegTune: Structured and Fine-Grained Control for Song Generation

2026-05-31

AI Summary

Existing neural approaches to song generation struggle to model time-varying musical attributes and lack fine-grained control over structural and dynamic aspects. To address these limitations, this work proposes SegTune, a diffusion Transformer-based framework that introduces a segment-level musical prompting mechanism. By integrating global style prompts with an LLM-driven lyric duration predictor, SegTune enables structured, high-fidelity lyric-to-song synthesis with precise lyric–music alignment. The approach is supported by a newly curated, high-quality annotated dataset and novel evaluation metrics tailored to song generation. Experimental results demonstrate that SegTune significantly outperforms current baselines in both musical quality and controllability.

Abstract

Recent advances in neural song generation have enabled high-quality synthesis from lyrics and global textual prompts. However, most systems fail to model temporally varying attributes of songs, severely limiting fine-grained control over musical structure and dynamics. To address this, we propose SegTune, a Diffusion Transformer-based framework enabling structured and fine-grained controllability by allowing users or large language models (LLMs) to specify local musical descriptions aligned to song segments. These segment prompts are temporally broadcast to corresponding time windows, while global prompts ensure stylistic coherence. To support precise lyric-to-music alignment, we introduce an LLM-based duration predictor that autoregressively generates sentence-level timestamps in LyRiCs format. We further construct a large-scale data pipeline for high-quality song collection with aligned lyrics and prompts, and propose new metrics to evaluate segment alignment and vocal consistency. Experiments demonstrate that SegTune outperforms existing baselines in both musicality and controllability. Visit our project page (https://github.com/KlingAIResearch/SegTune) for codes and more generated songs.

Problem

• song generation
• fine-grained control
• temporal attributes
• musical structure
• lyric-to-music alignment

Innovation

• fine-grained control
• segment-level prompting
• diffusion transformer
• lyric-to-music alignment
• LLM-based duration prediction