Skeleton Motion Words for Unsupervised Skeleton-Based Temporal Action Segmentation

2025-08-06

AI Summary

Unsupervised temporal action segmentation for skeletal sequences remains unexplored; existing approaches either rely on costly manual annotations (supervised) or focus on video inputs, overlooking skeletons’ advantages in robustness and privacy preservation. This paper introduces the first unsupervised skeleton-based action segmentation framework. It employs a temporal autoencoder to learn disentangled joint-motion representations, partitions the latent sequence into non-overlapping blocks, and applies vector quantization to derive discrete “skeleton motion words” that explicitly encode action semantics. Segmentation is achieved end-to-end via clustering-driven learning. To our knowledge, this is the first work to incorporate motion words into skeleton action segmentation. Our method achieves state-of-the-art performance among unsupervised approaches on HuGaDB, LARa, and BABEL. The source code is publicly available.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Unsupervised & Self-Supervised Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Current state-of-the-art methods for skeleton-based temporal action segmentation are predominantly supervised and require annotated data, which is expensive to collect. In contrast, existing unsupervised temporal action segmentation methods have focused primarily on video data, while skeleton sequences remain underexplored, despite their relevance to real-world applications, robustness, and privacy-preserving nature. In this paper, we propose a novel approach for unsupervised skeleton-based temporal action segmentation. Our method utilizes a sequence-to-sequence temporal autoencoder that keeps the information of the different joints disentangled in the embedding space. Latent skeleton sequences are then divided into non-overlapping patches and quantized to obtain distinctive skeleton motion words, driving the discovery of semantically meaningful action clusters. We thoroughly evaluate the proposed approach on three widely used skeleton-based datasets, namely HuGaDB, LARa, and BABEL. The results demonstrate that our model outperforms the current state-of-the-art unsupervised temporal action segmentation methods. Code is available at https://github.com/bachlab/SMQ .

Problem

• Unsupervised skeleton-based temporal action segmentation without annotated data
• Disentangling joint information in embedding space for better segmentation
• Quantizing latent sequences to discover meaningful action clusters

Innovation

• Sequence-to-sequence autoencoder for skeleton disentanglement
• Non-overlapping patches quantized into motion words
• Unsupervised action clustering via semantic motion words