Uni4D: A Unified Self-Supervised Learning Framework for Point Cloud Videos

2025-04-07

AI Summary

This work addresses two key bottlenecks in self-supervised representation learning for point cloud videos: poor generalization of motion modeling and difficulty in disentangling 4D spatiotemporal features. To this end, we propose SD-MAE—the first self-disentangling masked autoencoder framework for point cloud video understanding. Methodologically, SD-MAE introduces implicit motion representation learning in latent space, circumventing biases inherent in explicit motion estimation; it further designs a synergistic mechanism between implicit tokens and geometric tokens to enable multi-granularity disentanglement and reconstruction of hierarchical 4D features. Compared to conventional MAE-based approaches, SD-MAE significantly bridges the representational gap for 4D data, enhancing transferability of pre-trained representations. Extensive experiments demonstrate state-of-the-art performance across five benchmarks—including HOI4D—outperforming both existing self-supervised and fully supervised methods. Notably, SD-MAE achieves a 3.8% absolute improvement in action segmentation accuracy on HOI4D.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Representation Learning for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Point cloud video representation learning is primarily built upon the masking strategy in a self-supervised manner. However, the progress is slow due to several significant challenges: (1) existing methods learn the motion particularly with hand-crafted designs, leading to unsatisfactory motion patterns during pre-training which are non-transferable on fine-tuning scenarios. (2) previous Masked AutoEncoder (MAE) frameworks are limited in resolving the huge representation gap inherent in 4D data. In this study, we introduce the first self-disentangled MAE for learning discriminative 4D representations in the pre-training stage. To address the first challenge, we propose to model the motion representation in a latent space. The second issue is resolved by introducing the latent tokens along with the typical geometry tokens to disentangle high-level and low-level features during decoding. Extensive experiments on MSR-Action3D, NTU-RGBD, HOI4D, NvGesture, and SHREC'17 verify this self-disentangled learning framework. We demonstrate that it can boost the fine-tuning performance on all 4D tasks, which we term Uni4D. Our pre-trained model presents discriminative and meaningful 4D representations, particularly benefits processing long videos, as Uni4D gets $+3.8%$ segmentation accuracy on HOI4D, significantly outperforming either self-supervised or fully-supervised methods after end-to-end fine-tuning.

Problem

• Learning motion representation in latent space for better transferability
• Resolving 4D data representation gap using disentangled tokens
• Improving fine-tuning performance on diverse 4D tasks

Innovation

• Self-disentangled MAE for 4D representation learning
• Latent space motion modeling for better transferability
• Latent and geometry tokens for feature disentanglement