Multimodal Surface EMG Hand Gesture Recognition Using Query-Based Transformers for Prosthetic Control

2026-07-24

AI Summary

This study addresses the performance degradation of existing surface electromyography (sEMG)-based gesture recognition methods as the number of gestures increases, their limited capacity to model long-range temporal dependencies, and the lack of effective fusion with other physiological modalities such as inertial and eye-tracking signals. To overcome these challenges, this work proposes EMG-CrossFormer, an end-to-end hybrid convolutional–Transformer architecture that, for the first time, integrates learnable gesture queries into a Transformer framework for multimodal sEMG recognition. The model employs cascaded cross-attention mechanisms to jointly capture local and global spatiotemporal features and supports flexible input from arbitrary modalities. Experiments demonstrate that EMG-CrossFormer achieves 79.16% accuracy using sEMG alone across four NinaPro datasets, which further improves to 92.79% when inertial signals are incorporated—significantly outperforming six state-of-the-art baselines.

Abstract

Hand gesture recognition via surface electromyography (sEMG) is fundamental to prosthetic control. In this field, deep learning approaches have become the gold standard. However, current architectures struggle to scale; model performance typically decreases as the number of hand movements increases. Performance degradation is tied to the increased statistical complexity of decoding expanded gesture sets and compounded by the limitations of state-of-the-art methods, which primarily rely on low-latency unimodal convolutional architectures. Convolutions operate locally, limiting model's ability to capture long-range sequential patterns. Unimodal setups cannot leverage complementary information from coordinated signals characterizing movement execution, such as inertial and eye-tracking data. These limitations motivate architectures that integrate local and global features across multimodal physiological sequences. To bridge this gap, this study introduces EMG-CrossFormer, an end-to-end hybrid convolutional-transformer for seamless multimodal integration. EMG-CrossFormer combines representations from an arbitrary number of unimodal encoders through cascaded cross-attention fusion layers, and decodes the fused representations using learnable gesture queries. EMG-CrossFormer was evaluated on four NinaPro datasets (DB2, DB3, DB7, and DB10) and benchmarked against six state-of-the-art models using an increasing number of modalities. Using only sEMG, EMG-CrossFormer achieved mean accuracies of 72.33%, 52.48%, 79.16%, and 73.49% on DB2, DB3, DB7, and DB10, respectively. Incorporating inertial signals improved performance to 90.66%, 80.40%, 92.79%, and 92.06%. These results show that joint local-global feature modeling improves sEMG-only decoding and that multimodal fusion substantially amplifies this benefit, underscoring the value of both design principles for complex hand gesture recognition.

Problem

• surface electromyography
• hand gesture recognition
• multimodal fusion
• prosthetic control
• scalability

Innovation

• multimodal fusion
• query-based transformer
• cross-attention
• hand gesture recognition
• sEMG