AGIR: Assessing 3D Gait Impairment with Reasoning based on LLMs

2025-03-23

AI Summary

Current 3D gait assessment for neurodegenerative diseases suffers from subjectivity and lacks clinical interpretability in deep learning models. Method: We propose an interpretable, end-to-end gait scoring framework that maps raw 3D motion sequences directly to MDS-UPDRS gait subscore ratings. Our approach innovatively integrates Vector Quantized Variational Autoencoder (VQ-VAE)-based motion tokenization with Large Language Model (LLM)-driven Chain-of-Thought (CoT) reasoning. We construct the first Parkinson’s multimodal gait dataset annotated with pathology-informed reasoning traces and introduce a two-stage supervised fine-tuning strategy to jointly optimize motion-to-text and text-to-motion generation alongside clinical logic inference. Contribution/Results: Our method achieves significant improvements over existing state-of-the-art approaches on public benchmarks, delivering both high predictive accuracy and clinically compliant, step-by-step interpretability—demonstrating robustness and real-world clinical applicability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Assessing gait impairment plays an important role in early diagnosis, disease monitoring, and treatment evaluation for neurodegenerative diseases. Despite its widespread use in clinical practice, it is limited by subjectivity and a lack of precision. While recent deep learning-based approaches have consistently improved classification accuracies, they often lack interpretability, hindering their utility in clinical decision-making. To overcome these challenges, we introduce AGIR, a novel pipeline consisting of a pre-trained VQ-VAE motion tokenizer and a subsequent Large Language Model (LLM) fine-tuned over pairs of motion tokens and Chain-of-Thought (CoT) reasonings. To fine-tune an LLM for pathological gait analysis, we first introduce a multimodal dataset by adding rationales dedicated to MDS-UPDRS gait score assessment to an existing PD gait dataset. We then introduce a two-stage supervised fine-tuning (SFT) strategy to enhance the LLM's motion comprehension with pathology-specific knowledge. This strategy includes: 1) a generative stage that aligns gait motions with analytic descriptions through bidirectional motion-description generation, 2) a reasoning stage that integrates logical Chain-of-Thought (CoT) reasoning for impairment assessment with UPDRS gait score. Validation on an existing dataset and comparisons with state-of-the-art methods confirm the robustness and accuracy of our pipeline, demonstrating its ability to assign gait impairment scores from motion input with clinically meaningful rationales.

Problem

• Assessing gait impairment lacks precision and interpretability in clinical practice.
• Deep learning methods improve accuracy but lack clinical interpretability for decision-making.
• AGIR combines motion tokenizer and LLM for interpretable gait impairment assessment.

Innovation

• VQ-VAE motion tokenizer for gait analysis
• LLM fine-tuned with Chain-of-Thought reasoning
• Two-stage supervised fine-tuning strategy