SegTune: Structured and Fine-Grained Control for Song Generation

2025-10-21

AI Summary

Existing song generation methods struggle to model the temporal evolution of musical structure and dynamics, resulting in insufficient fine-grained controllability. To address this, we propose a non-autoregressive song generation framework. Our method introduces three key innovations: (1) a paragraph-level temporal alignment control mechanism that explicitly coordinates lyric paragraphs with musical structure; (2) an LLM-driven duration prediction approach for LRC-formatted lyrics, enabling construction of high-quality temporally aligned training data; and (3) a multi-component prompting strategy integrating local-global text prompts, time-broadcasted conditioning injection, and multi-granularity prompt fusion. Experiments demonstrate significant improvements over baselines in paragraph alignment accuracy, vocal attribute consistency, and musical coherence. The framework enables high-fidelity, structurally controllable song synthesis. Furthermore, we introduce novel evaluation metrics to quantitatively assess controllability and cross-modal consistency.

Technology Category

Natural Language Processing: GenerationMachine Learning: Large Multimodal Models (LMMs)Planning, Routing, and Scheduling: Planning with Language Models

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSearch and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAGEconomics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent advancements in song generation have shown promising results in generating songs from lyrics and/or global text prompts. However, most existing systems lack the ability to model the temporally varying attributes of songs, limiting fine-grained control over musical structure and dynamics. In this paper, we propose SegTune, a non-autoregressive framework for structured and controllable song generation. SegTune enables segment-level control by allowing users or large language models to specify local musical descriptions aligned to song sections.The segmental prompts are injected into the model by temporally broadcasting them to corresponding time windows, while global prompts influence the whole song to ensure stylistic coherence. To obtain accurate segment durations and enable precise lyric-to-music alignment, we introduce an LLM-based duration predictor that autoregressively generates sentence-level timestamped lyrics in LRC format. We further construct a large-scale data pipeline for collecting high-quality songs with aligned lyrics and prompts, and propose new evaluation metrics to assess segment-level alignment and vocal attribute consistency. Experimental results show that SegTune achieves superior controllability and musical coherence compared to existing baselines. See https://cai525.github.io/SegTune_demo for demos of our work.

Problem

• Modeling temporally varying song attributes for fine-grained control
• Enabling segment-level musical control through local descriptions
• Achieving precise lyric-to-music alignment with duration prediction

Innovation

• Non-autoregressive framework for structured song generation
• Segment-level control via temporally broadcasted prompts
• LLM-based duration predictor for precise lyric alignment