Technology Category
Intelligent Robots: Multimodal Perception & Sensor FusionMachine Learning: Multimodal LearningComputer Vision: Multi-modal Vision
Joint Transcription of Acoustic Guitar Strumming Directions and Chords
🤖 AI Summary
Existing approaches to joint automatic transcription of guitar strumming direction and chord progressions are hampered by severe scarcity of real-world annotated data, resulting in suboptimal performance. To address this, we introduce the first multimodal dataset integrating authentic strumming motion signals captured from smartwatches with synthetically generated audio, and propose an end-to-end CRNN model that jointly detects strumming events, classifies strumming direction (up/down), and recognizes chords from monaural audio input. Our key innovations are: (1) the first incorporation of time-synchronized motion sensor signals to augment audio-based analysis; and (2) a real–synthetic co-enhancement data paradigm. Experiments demonstrate significant improvements over strong baselines—achieving absolute accuracy gains of 5.3%–8.7% on both strumming detection and chord recognition—establishing a new benchmark and high-quality resource for rhythm guitar MIR research.
Technology Category
Application Category
📝 Abstract
Automatic transcription of guitar strumming is an underrepresented and challenging task in Music Information Retrieval (MIR), particularly for extracting both strumming directions and chord progressions from audio signals. While existing methods show promise, their effectiveness is often hindered by limited datasets. In this work, we extend a multimodal approach to guitar strumming transcription by introducing a novel dataset and a deep learning-based transcription model. We collect 90 min of real-world guitar recordings using an ESP32 smartwatch motion sensor and a structured recording protocol, complemented by a synthetic dataset of 4h of labeled strumming audio. A Convolutional Recurrent Neural Network (CRNN) model is trained to detect strumming events, classify their direction, and identify the corresponding chords using only microphone audio. Our evaluation demonstrates significant improvements over baseline onset detection algorithms, with a hybrid method combining synthetic and real-world data achieving the highest accuracy for both strumming action detection and chord classification. These results highlight the potential of deep learning for robust guitar strumming transcription and open new avenues for automatic rhythm guitar analysis.
Problem
Research questions and friction points this paper is trying to address.
Transcribing guitar strumming directions and chords from audio
Overcoming dataset limitations for accurate strumming analysis
Improving chord classification and strumming detection using deep learning
Innovation
Methods, ideas, or system contributions that make the work stand out.
Multimodal approach with smartwatch motion sensor
CRNN model for strumming and chord detection
Hybrid synthetic-real dataset enhances accuracy
🔎 Similar Papers
No similar papers found.
