SKY-Piano: A Multimodal Piano Performance Dataset

2026-07-29

AI Summary

This study addresses the lack of high-quality, multimodal synchronized datasets in piano performance research by introducing and open-sourcing the SKY-Piano dataset. Comprising 11 hours of performances by both professional and amateur pianists, it is the first to jointly integrate high-fidelity full-body motion capture (kinematics), multi-view video, audio, MIDI, and MusicXML scores with precise temporal alignment. The work contributes an interactive multimodal browsing tool, a fingering pseudo-labeling model leveraging MIDI and motion data, and demonstrates the dataset’s utility through MIDI-to-motion generation experiments. SKY-Piano thus provides robust support for research in music information retrieval, human–AI collaborative performance, and multimodal alignment.

Abstract

Music information retrieval research on piano performance increasingly involves diverse modalities of data and annotations beyond audio and MIDI. We present SKY-Piano, a multimodal piano performance dataset that includes 11 hours of performance recordings of motion, multi-view video, audio, MIDI from 7 professional and 12 amateur pianists along with MusicXML scores. The performance pieces were selected considering playing technique, difficulty, and performer expertise on a shared core repertoire. The motion data include both hand and body motion, released in both flagged form, where samples lost to marker occlusion are marked as unreliable, and imputed form, where those gaps are reconstructed, together with Visual3D body-segment kinematics and other time-synchronized modalities. To easily browse different modalities of data at a glance, we provide an interactive web browser. In addition, we developed a fingering annotation model and tool for deriving pseudo fingering annotations from the MIDI and motion data. Lastly, we present MIDI-to-motion generation through a fine-tuning experiment as a use case of the dataset.

Problem

• multimodal dataset
• piano performance
• music information retrieval
• motion capture
• fingering annotation

Innovation

• multimodal dataset
• piano performance
• motion capture
• fingering annotation
• MIDI-to-motion generation