P-MUSE: Prompt-MIDI-Optional Model for Unified Instrumental Music Synthesis and Editing

2026-08-03

AI Summary

This work addresses the longstanding challenge in MIDI-to-music synthesis of balancing the flexibility of unaligned MIDI inputs with the precise control afforded by aligned ones. To this end, the authors propose P-MUSE, a unified framework that integrates music generation and local editing through a fill-in-the-blank modeling paradigm and supports optional Prompt-MIDI conditioning. Key innovations include the first fusion of conditional generation with in-context learning, a phase-aware classifier-free guidance schedule, a Tail-Drop strategy, and a multi-stage curriculum learning approach enhanced by cross-modal alignment. Experiments across piano, guitar, bass, and drums demonstrate that P-MUSE significantly improves MIDI fidelity and timbral accuracy. The study also introduces and open-sources the first comprehensive benchmark supporting multiple prompting modes and instrument types, thereby advancing evaluation standards in the field.

Abstract

MIDI-to-Music system renders the melody and rhythm of a target MIDI sequence into musical segment while cloning instrument timbre from a prompt recording. Existing systems typically adopt one of two distinct paradigms: conditional generation with prompt audio alone, which remains applicable when aligned prompt MIDI is unavailable, and In-Context Learning with paired prompt audio and MIDI, which exploits cross-modal alignment for stronger control on MIDI following and timbre similarity. We introduce P-MUSE, an instrumental MIDI-to-Music framework that unifies both paradigms via a multi-stage Curriculum-Learning supporting prompt-MIDI-optional inputs. P-MUSE further unifies music generation and local editing through a shared fill-in-the-middle formulation. Grounded in theoretical analysis and empirical study, we propose a phase-aware classifier-free guidance scheduling principle for Transcription-to-Audio systems, alongside a Tail-Drop strategy. Finally, to advance research in this field, we establish the first comprehensive benchmark, covering various prompt modes, generation/editing tasks, and four representative instruments: piano, guitar, bass, and drums. Demos are available at https://p-muse.github.io/.

Problem

• MIDI-to-Music
• instrumental music synthesis
• prompt-based generation
• music editing
• timbre cloning

Innovation

• Prompt-MIDI-Optional
• Unified Music Synthesis and Editing
• Fill-in-the-Middle
• Phase-Aware Classifier-Free Guidance
• Curriculum Learning