Multi-Task Multi-Frame Visual Piano Transcription

2026-08-04

AI Summary

Existing vision-based piano transcription methods exhibit limited performance in predicting note offsets and dynamics, largely due to their reliance on short video clips that hinder accurate capture of key release timing and expressive nuances. This work proposes V2N, a system featuring a shared temporal convolutional backbone with multi-task heads to jointly predict note onsets, offsets, durations, and time-varying velocities. By incorporating full-frame supervision signals, the model enables end-to-end training and, for the first time, achieves complete visual piano transcription. Leveraging long-range temporal context and multi-task learning, V2N significantly improves offset and velocity prediction accuracy while also enhancing onset detection. The method sets new state-of-the-art results on the PianoVAM and R3 datasets, substantially narrowing the performance gap between onset and offset prediction and reporting, for the first time, quantitative metrics for velocity estimation using purely visual approaches.

Abstract

Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.

Problem

• Visual Piano Transcription
• note offset
• velocity estimation
• multi-task learning
• temporal context

Innovation

• Multi-Task Learning
• Visual Piano Transcription
• Per-Frame Supervision
• Temporal Context
• Note-Level Velocity