Exploring Adaptive Masked Reconstruction for Self-Supervised Skeleton-Based Action Recognition

2026-06-09

AI Summary

This work addresses the inefficiency and limited semantic focus of existing self-supervised action recognition methods based on masked skeleton reconstruction, which typically require reconstructing numerous spatiotemporal blocks. To overcome these limitations, we propose an Adaptive Masked Reconstruction (AMR) framework that introduces, for the first time in skeleton-based action recognition, a decoupled encoder–decoder architecture enabling efficient reconstruction of large-scale spatiotemporal blocks. Furthermore, we design an adaptive guidance module driven by motion information entropy to dynamically emphasize highly discriminative regions during training. Extensive experiments demonstrate that our approach significantly accelerates pretraining on the NTU RGB+D 60/120 and PKU-MMD datasets while achieving state-of-the-art performance on downstream action recognition tasks.

Abstract

Recently, masked skeleton reconstruction models have emerged as strong action representation learners, driving significant progress in self-supervised skeleton-based action recognition. However, existing state-of-the-art methods must predict an exceedingly large number of spatiotemporal patches, significantly prolonging training time. Besides, by treating all spatiotemporal regions equally during reconstruction, these models are distracted from learning the critical motion patterns that underlie action semantics. To address these challenges, we propose Adaptive Masked Reconstruction (AMR), a faster and stronger pre-training framework. We first decouple the decoder from the encoder, enabling flexible prediction of larger spatiotemporal patches and dramatically reducing reconstruction complexity. Given that larger patches contain more complex information, which is challenging to predict and consequently degrades performance, we accordingly introduce an adaptive guidance module. This module identifies regions of high motion informativeness, guiding the model to focus on the most discriminative parts of each patch and alleviating reconstruction difficulty. Experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD datasets demonstrate that AMR not only accelerates pre-training substantially but also improves downstream recognition accuracy, surpassing current state-of-the-art approaches.

Problem

• masked skeleton reconstruction
• self-supervised learning
• action recognition
• spatiotemporal patches
• motion patterns

Innovation

• Adaptive Masked Reconstruction
• Self-Supervised Learning
• Skeleton-Based Action Recognition
• Spatiotemporal Patch Prediction
• Motion Informativeness