Profy: Interpretable Visualization of Expertise-Dependent Motor Skills Toward Supporting Piano Practice

2026-06-09

AI Summary

This work addresses the long-standing lack of fine-grained, actionable feedback in piano practice, where traditional approaches offer only global scores that inadequately guide specific improvements. To overcome this limitation, the authors propose Profy, a system that leverages weakly supervised learning to automatically extract temporally aligned highlight segments from holistic expert and amateur performance ratings, generating interpretable localized feedback without requiring frame-level annotations. Profy integrates synchronized 1 kHz keystroke motion and audio data, employing temporal resampling modeling and evidence-score visualization to enable dynamic playback, looping, and frame-by-frame review. Experiments on 20 amateur performances demonstrate strong alignment between Profy’s highlighted regions and expert annotations (Pearson r = 0.61, ROC-AUC = 0.75), effectively directing learners’ focus and surpassing the constraints of conventional global assessment.

Abstract

The quality of piano performance depends on nuanced timing, articulation, and dynamic control, but practice feedback is often summary-based and hard to act on. We introduce Profy, a weakly supervised system that learns from take-level labels derived from aggregated listener ratings (expert-labeled vs. amateur-labeled) to produce time-aligned highlights for review during piano practice. We collected synchronized 1 kHz key-motion and audio from 73 pianists and used 1,083 valid takes for modeling and evaluation. The model outputs clip-level predictions together with evidence scores on a shared resampled model time base for visualization. On 20 amateur clips from short technique studies annotated by 21 expert pianists, the displayed highlight score aligns with passages that expert pianists marked for review despite training without localized labels (Pearson r=0.61, ROC-AUC 0.75). Rather than summarizing a take with a single global score, Profy helps learners decide where to inspect next by supporting scrubbing, looping, and focused replay of time-localized passages associated with expert-amateur differences.

Problem

• piano practice
• motor skills
• performance feedback
• expertise-dependent
• time-aligned highlights

Innovation

• weakly supervised learning
• interpretable visualization
• time-aligned feedback
• expertise-dependent motor skills
• piano practice support