SketchSong: Hierarchical Song Generation with Sketch Planning and Fine-Grained Multi-Track Modeling

2026-06-02

AI Summary

Existing song generation systems often suffer from abrupt section transitions, insufficient dynamic range, and monotonous arrangements due to the absence of explicit structural planning and fine-grained multi-track modeling. This work proposes a hierarchical generative framework that first predicts high-level sketch tokens from compressed audio representations to outline the global song structure, then conditions fine-grained audio generation on this sketch. The approach explicitly models four distinct tracks—vocals, bass, drums, and other instruments—to accurately capture the role and interaction of each musical part. By integrating hierarchical architecture, sketch-guided generation, and parallel multi-track modeling, the method substantially enhances structural coherence and arrangement richness. It outperforms baseline models on both objective metrics and human listening tests, achieving generation quality comparable to strong open-source systems—even without lyric alignment or preference-based optimization.

Abstract

Recent song generation systems can synthesize realistic audio, yet generating complete songs remains challenging for two reasons. First, explicit song-level arrangement planning remains limited in existing methods, so models often need to organize overall arrangement development while generating low-level audio details. This often leads to incoherence in arrangements, such as weak section transitions and limited dynamic progression. Second, coarse modeling of different musical parts obscures their distinct roles and interactions, limiting arrangement richness of generated songs. In this paper, we present SketchSong, a hierarchical song generation framework that addresses these issues through song-level sketch planning and fine-grained multi-track modeling. Along the temporal dimension, SketchSong first predicts a compact sequence of high-level sketch tokens derived from compressed audio representations, and then generates audio tokens conditioned on these sketches. This coarse-to-fine process gives the model an explicit arrangement plan before detailed audio generation. Along the track dimension, SketchSong explicitly models four tracks, i.e., vocals, bass, drums and other instruments. This enables the model to capture the roles and interactions of different musical parts more precisely. Experiments on song generation benchmarks show that SketchSong consistently outperforms our baseline on both objective metrics and human listening tests. Despite not employing additional post-training for preference optimization such as lyrics and text-prompt alignments, SketchSong achieves competitive results against strong, post-trained open-source systems, demonstrating the effectiveness of our overall design.

Problem

• song generation
• arrangement planning
• multi-track modeling
• coherence
• musical structure

Innovation

• hierarchical generation
• sketch planning
• multi-track modeling
• song arrangement
• coarse-to-fine generation