Music-JEPA: Learning a World Model of Sound from Action

2026-07-24

AI Summary

This work proposes a music understanding and generation framework based on an action-conditional world model, formulating piano performance as a system where pianoroll sequences serve as actions and audio signals as states. It introduces the Joint Embedding Predictive Architecture (JEPA) to the music domain for the first time, leveraging self-supervised learning on offline audio–pianoroll paired data to predict future audio states without requiring environmental interaction, thereby capturing the mapping between sonic outcomes and performance actions. The learned representations effectively support downstream tasks such as beat tracking, composer identification, and key estimation, and enable high-quality automatic piano transcription through action planning, demonstrating JEPA’s potential for music representation learning.

Abstract

Joint Embedding Predictive Architectures (JEPA) have recently emerged as a paradigm for learning world models by predicting latent representations, offering a promising direction for self-supervised learning. While initial attempts have applied JEPA to the music domain, it remains unclear how such frameworks can naturally support the formation of a world model for music. In this work, we propose to learn a world model of piano sound using JEPA by framing music as an action-conditioned system: the audio is treated as the state, and the pianoroll as the instrument action. Given a current audio state and an action, the model predicts the resulting future audio state, mirroring how humans learn musical sound through interaction. The model is trained in a fully offline setting using paired audio-pianoroll data, without environment interaction. Experiments show that the learned model captures the relationships between musical actions and their resulting sound. The resulting representations support downstream tasks, including beat tracking, composer identification, and key estimation, and enable piano transcription via planning, by searching for actions that best explain a target sound.

Problem

• world model
• music
• action-conditioned
• self-supervised learning
• sound representation

Innovation

• Joint Embedding Predictive Architecture
• world model
• action-conditioned audio prediction
• self-supervised music learning
• piano transcription via planning