MR-STGN: Multi-Residual Spatio Temporal Graph Network Using Attention Fusion for Patient Action Assessment

2023-09-27

IEEE International Workshop on Multimedia Signal Processing

AI Summary

To address the subjectivity and inefficiency of clinical motor assessments relying on manual scoring, this paper proposes an end-to-end automated scoring system. Methodologically, we design a multi-residual spatio-temporal graph convolutional network (ST-GCN) that jointly models multi-granular spatio-temporal dynamics of 3D skeletal joint positions and Euler angles. We further introduce a novel joint-pose dual-stream attention fusion mechanism to enable interpretable, region-level feature weighting. Our contributions are threefold: (1) the first integration of multi-residual architecture and dual-stream skeletal encoding into medical motor assessment; (2) state-of-the-art performance on the UI-PRMD dataset, achieving a 5.2% AUC improvement over prior methods; and (3) high accuracy in critical motion recognition (F1 = 91.4%) with real-time inference capability for precise score prediction.

Abstract

Accurate assessment of patient actions plays a crucial role in healthcare as it contributes significantly to disease progression monitoring and treatment effectiveness. However, traditional approaches to assess patient actions often rely on manual observation and scoring, which are subjective and time-consuming. In this paper, we propose an automated approach for patient action assessment using a Multi-Residual Spatio Temporal Graph Network (MR-STGN) that incorporates both angular and positional 3D skeletons. The MR-STGN is specifically designed to capture the spatio-temporal dynamics of patient actions. It achieves this by integrating information from multiple residual layers, with each layer extracting features at distinct levels of abstraction. Furthermore, we integrate an attention fusion mechanism into the network, which facilitates the adaptive weighting of various features. This empowers the model to concentrate on the most pertinent aspects of the patient's movements, offering precise instructions regarding specific body parts or movements that require attention. Ablation studies are conducted to analyze the impact of individual components within the proposed model. We evaluate our model on the UI-PRMD dataset demonstrating its performance in accurately predicting real-time patient action scores, surpassing state-of-the-art methods.

Problem

• Automated patient action assessment
• Spatio-temporal dynamics of actions
• Attention fusion for adaptive weighting

Innovation

• Multi-Residual Spatio Temporal Graph
• Attention Fusion Mechanism
• 3D Skeleton Feature Extraction