ImprovNet: Generating Controllable Musical Improvisations with Iterative Corruption Refinement

2025-02-06

AI Summary

In symbolic music, controllable and expressive performance-level style transfer—particularly for niche genres like jazz—is hindered by data scarcity and the absence of unified multi-task models. This paper introduces the first Transformer-based model capable of cross- and intra-genre improvisation generation, melody harmonization, short-prompt continuation, and phrase completion within a single architecture. We propose a novel self-supervised iterative corruption–refinement training paradigm, integrating music event encoding with beat-aware positional embeddings to enable fine-grained control over stylistic intensity and structural fidelity. Our model significantly outperforms Anticipatory Music Transformer on continuation and completion tasks. In user studies, 79% of participants correctly identified generated jazz improvisations. Both objective metrics and subjective evaluations confirm that outputs exhibit high musical coherence and faithful preservation of original structural characteristics.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Deep learning has enabled remarkable advances in style transfer across various domains, offering new possibilities for creative content generation. However, in the realm of symbolic music, generating controllable and expressive performance-level style transfers for complete musical works remains challenging due to limited datasets, especially for genres such as jazz, and the lack of unified models that can handle multiple music generation tasks. This paper presents ImprovNet, a transformer-based architecture that generates expressive and controllable musical improvisations through a self-supervised corruption-refinement training strategy. ImprovNet unifies multiple capabilities within a single model: it can perform cross-genre and intra-genre improvisations, harmonize melodies with genre-specific styles, and execute short prompt continuation and infilling tasks. The model's iterative generation framework allows users to control the degree of style transfer and structural similarity to the original composition. Objective and subjective evaluations demonstrate ImprovNet's effectiveness in generating musically coherent improvisations while maintaining structural relationships with the original pieces. The model outperforms Anticipatory Music Transformer in short continuation and infilling tasks and successfully achieves recognizable genre conversion, with 79% of participants correctly identifying jazz-style improvisations. Our code and demo page can be found at https://github.com/keshavbhandari/improvnet.

Problem

• Generating controllable musical improvisations
• Handling multiple music generation tasks
• Achieving expressive style transfer in symbolic music

Innovation

• Transformer-based architecture
• Self-supervised corruption-refinement training
• Iterative style transfer control