SRENet: Spectral Re-Entry Network for Point Cloud Action Recognition

2026-06-02

AI Summary

This work addresses the challenge of simultaneously modeling global motion context and fine-grained temporal dynamics in point cloud sequences, which suffer from structural irregularity and temporal inconsistency. To this end, the authors propose a novel frequency-domain approach to spatiotemporal representation learning for point cloud action recognition, introducing wavelet analysis and curriculum contrastive learning for the first time in this domain. The method explicitly captures multiscale spatiotemporal dynamics through a spectral decomposition block and a spectral re-entry block, integrating wavelet transforms, frequency-specific attention mechanisms, and a curriculum-scheduled contrastive loss. Extensive experiments demonstrate state-of-the-art performance on MSR-Action3D, NTU-RGB+D, and NTU-RGB+D120 benchmarks, validating the efficacy and superiority of frequency-domain modeling for point cloud action recognition.

Abstract

Recognizing human actions from point cloud sequences is critical for 3D perception driven applications such as autonomous driving and human-computer interaction. However, the irregular structure and temporal inconsistency of point clouds pose unique challenges for spatio-temporal representation learning, especially in capturing both global motion context and fine-grained temporal dynamics. We propose SRENet, a spectral-aware framework designed to explicitly learn both global context and fine-grained temporal dynamics of motion from a frequency perspective for action recognition. SRENet introduces a Spectral Decomposition Block (SDeBlock) that performs wavelet-based analysis along temporal and spatial axes, disentangling features into low- and high-frequency components with frequency-specific attention. To recover residual dynamics and re-align temporal frequency structures distorted during semantic fusion, a Spectral Re-entry Block (SReBlock) performs secondary temporal decomposition. Furthermore, a spectral-aware learning strategy is devised to enhance discriminability in both frequency subspaces via contrastive loss and a curriculum schedule that gradually shifts focus from low- to high-frequency spaces in line with coarse to detailed motion patterns. Extensive experiments on MSR-Action3D, NTU-RGBD and NTU-RGBD120 demonstrate that SRENet achieves state-of-the-art performance, validating the effectiveness of frequency modeling in point cloud-based action understanding.

Problem

• point cloud action recognition
• spatio-temporal representation learning
• temporal inconsistency
• global motion context
• fine-grained temporal dynamics

Innovation

• spectral decomposition
• wavelet-based analysis
• frequency-aware learning
• point cloud action recognition
• temporal dynamics