multimodal assessment

This study addresses the challenge of performing accurate, lab-free 3D human kinematic assessment during Activities of Daily Living (ADL) for telemedicine, sports science, and rehabilitation. We systematically benchmark monocular video-based and IMU-based approaches using state-of-the-art models—including MotionAGFormer, MotionBERT, MMPose (2D-to-3D), and NVIDIA BodyTrack—and unify evaluation via OpenSim inverse dynamics and Human3.6M joint-angle metrics. Results demonstrate that MotionAGFormer achieves the highest accuracy (RMSE = 9.27° ± 4.80°, MAE = 7.86° ± 4.18°, correlation r = 0.86 ± 0.15, R² = 0.67 ± 0.28), confirming the clinical feasibility of monocular video in real-world settings. We introduce a new benchmark for in-the-wild human motion capture, explicitly characterizing the trade-offs among accuracy, cost, and deployment practicality between video and IMU modalities. This work provides empirical validation and methodological guidance for scalable, low-cost remote motion monitoring.

High-quality, multimodal motion data are critically needed for physical therapy and gait analysis, yet existing datasets suffer from high acquisition costs and poor generalizability. To address this, we introduce the first open-source, multimodal dataset specifically designed for rehabilitation assessment. It comprises synchronized inertial measurement unit (IMU) data (9 channels) and optical motion capture data (68 markers) from 19 participants performing 12 standardized rehabilitation and gait tasks. Our key contributions include: (i) millisecond-level temporal synchronization between IMU and optical data; (ii) IMU orientation calibration within a standardized anatomical coordinate system; (iii) subject-specific OpenSim model–driven inverse kinematics outputs; and (iv) comprehensive temporal annotations and clinical expert ratings for all movements. We publicly release preprocessing code, validation tools, and an interactive visualization platform. This resource significantly enhances reproducibility and generalizability in movement quality assessment, temporal segmentation, and biomechanical modeling.

Existing vision/audio-based multimodal motion understanding methods struggle to model 3D dynamic forces and torques; while IMUs offer lightweight, privacy-preserving advantages, their utility for long-term, real-time motion capture (MoCap) and online analysis is hindered by wireless transmission instability, sensor noise, and drift. This paper introduces the first LLM-driven inertial motion instruction framework. It proposes jitter-suppressed inertial token representations to enable noise-robust temporal modeling and semantic motion parsing. The framework integrates a lightweight IMU array, a jitter-aware encoder, an LLM-enhanced motion–language alignment architecture, and edge–cloud collaborative streaming inference. Evaluated in real-world settings, it achieves 92.3% action intention recognition accuracy with sub-80 ms latency, reduces drift error by 67%, and supports 12-hour calibration-free continuous MoCap with real-time behavioral feedback.

Current wearable inertial measurement unit (IMU) deployments rely on empirical conventions, lacking systematic evaluation and optimization. Method: We propose W2W, a simulation-driven framework featuring a high-fidelity, 512-point human surface model that synthesizes task-specific IMU signals. It enables fine-grained, cross-task utility quantification—e.g., for pose estimation and activity recognition—by integrating motion-capture-driven signal modeling with multimodal real-world data validation. Contribution/Results: W2W is the first to identify task-consistent high-value sensing regions, challenging the fixed-layout paradigm of commercial devices. Experiments demonstrate strong rank correlation (r > 0.92) between simulated and empirical performance, and uncover several non-canonical placements—overlooked by conventional approaches—that yield superior accuracy. W2W establishes the first open-source, reproducible, simulation-based paradigm for task-adaptive and scalable IMU placement design.

Multimodal human activity recognition in natural scenes faces challenges in fusing asynchronous, heterogeneous sensor data (video, audio, RFID) and lacks interpretability. Method: This paper proposes a reproducible multimodal fusion framework that jointly addresses temporal alignment, feature standardization, and semantic parsing; supports early, late, and hybrid fusion strategies; and incorporates sparse RFID signals to enhance discriminative capability. Interpretability is achieved via t-SNE visualization, attention heatmaps, and waveform superposition for modality-wise contribution analysis. Contribution/Results: The work presents the first systematic evaluation of multimodal synergy in temporal consistency and discriminability. Experiments show late fusion achieves optimal performance; integrating RFID improves classification accuracy by over 50% and significantly boosts macro-averaged ROC-AUC, demonstrating the framework’s dual advantages in accuracy and interpretability.

This study addresses the limited accuracy of generic kernel models in wearable IMU-based gait estimation caused by inter-individual variability. To overcome this, we propose a personalized kernel regression method based on sparse calibration. By constructing a kinematic reference library, the approach requires only minimal calibration data across three walking speeds. It combines user-specific baselines with kinematic deviations extracted via principal component analysis (PCA) to generate personalized kernel functions, effectively eliminating the reliance on extensive individual training data typical of conventional methods. Offline experiments demonstrate that the proposed method significantly reduces estimation errors for both gait phase and velocity. Furthermore, embedded online validation confirms its feasibility for real-time execution, providing an efficient solution for personalized gait monitoring.

This study addresses the unreliability of sparse inertial pose estimation with consumer-grade IMUs caused by firmware discrepancies, wearing variations, and signal drift. To tackle these challenges, we propose a channel-level reliability gating fusion method that adaptively learns trust weights for each sensor channel via a temporal gating network. The model is optimized through synthetic data pretraining combined with an auxiliary reliability objective function. Furthermore, we construct a benchmark dataset comprising 35 recording sessions across head-mounted earphones and foot-worn smart insoles. Experimental results demonstrate that the proposed approach achieves state-of-the-art accuracy (69.4 mm) under both clean data conditions and simulated failure scenarios, effectively suppressing sensor bias and enabling dropout detection.

This work addresses the challenges of fusing multi-source heterogeneous IMU data and insufficient long-term contextual modeling by proposing a tri-spectral fusion framework. The framework introduces adaptive filtering mechanisms in the Fourier, graph Fourier, and wavelet domains to effectively integrate pose, motion, and contextual information. It innovatively constructs a dynamic heterogeneous graph that combines adaptive complementary filtering, graph Fourier transforms, wavelet-based frequency selection, and timestamp-aware graph aggregation to suppress noise and redundancy while enhancing multimodal fusion and long-range dependency modeling. Evaluated on ten benchmark datasets, the proposed method significantly outperforms existing approaches, achieving state-of-the-art performance.

This study addresses the lack of systematic comparisons among sensor fusion methods under a unified multimodal human activity recognition benchmark. For the first time, it conducts a head-to-head evaluation of seven mainstream fusion strategies—including gated fusion and late concatenation—on the HARMES dataset, leveraging IMU, audio, and humidity signals to recognize 15 categories of daily activities. The experiments employ deep learning architectures with leave-one-participant-out cross-validation. Results demonstrate that gated multimodal fusion achieves the best performance, attaining a macro F1-score of 0.82, which represents a 6-percentage-point improvement over baseline methods, thereby confirming its effectiveness and superiority in multimodal activity recognition.

GaitVista通过轻量级门控和多模态数据融合，提高步态评估的可靠性和准确性，减少误差，实现更广泛的步态恢复追踪。