MyoSem: Aligning Electromyography to Natural-Language Action Semantics for Hand Action Understanding

2026-05-29

AI Summary

This work proposes MyoSem, a novel framework that reimagines hand gesture recognition from electromyography (EMG) signals as a language-mediated semantic retrieval task rather than a fixed-class classification problem. MyoSem establishes a queryable bidirectional semantic alignment between EMG signals and natural language descriptions of hand movements by constructing a shared embedding space through multi-perspective action semantics. It integrates activation-aware EMG encoding with semantic alignment mechanisms, synergistically combining signal processing and language embedding techniques. Experiments on the EMG2Pose and NinaPro datasets demonstrate superior performance in bidirectional retrieval tasks and strong generalization capabilities across unseen users, novel gesture categories, and even amputee subjects. This paradigm shift enables flexible, language-driven querying of hand actions and advances EMG-based gesture understanding beyond rigid categorical boundaries.

Abstract

Electromyography (EMG) directly reflects muscle activation and is a key sensing modality for gesture recognition, prosthetic control, and wearable interaction. Existing EMG methods, however, commonly formulate hand action understanding as classification over fixed labels, making it difficult to support querying, retrieval, and generalization based on action descriptions. We present MyoSem, an EMG--action semantic alignment framework that maps low-level EMG signals into a shared semantic space constructed from multi-view action descriptions. MyoSem combines multi-view action-semantic construction, activation-aware EMG encoding, and semantic query alignment, enabling bidirectional retrieval between EMG signals and text descriptions. We systematically evaluate MyoSem on EMG2Pose and NinaPro-series datasets. Results show that MyoSem performs well on EMG--text bidirectional retrieval, generally outperforms most baselines, and shows favorable generalization to unseen users, held-out action classes, and amputee-user transfer scenarios. Ablations and visualizations further validate the effectiveness of each module. Overall, MyoSem advances EMG-based hand action understanding from fixed-label recognition toward queryable bidirectional semantic retrieval, providing a new modeling paradigm for language-mediated EMG action understanding.

Problem

• Electromyography
• Hand Action Understanding
• Semantic Retrieval
• Natural Language
• Gesture Recognition

Innovation

• EMG--text alignment
• semantic retrieval
• action semantics
• bidirectional mapping
• language-mediated EMG understanding