CoCoDiff: Diversifying Skeleton Action Features via Coarse-Fine Text-Co-Guided Latent Diffusion

2025-04-30

AI Summary

To address insufficient feature diversity in skeleton-based action recognition and the semantic inconsistency and computational inefficiency of existing data augmentation methods, this paper proposes CoCoDiff—a plug-and-play training module that generates semantically consistent features in the latent space. Its core innovation lies in the first-ever diffusion mechanism jointly conditioned on coarse-grained (action category) and fine-grained (joint motion description) textual prompts, integrated with LLM-driven multi-granularity text encoding, spatiotemporal skeleton feature modeling, and unified conditional control. Crucially, CoCoDiff enhances feature diversity at zero inference overhead. Extensive experiments demonstrate state-of-the-art performance on NTU RGB+D, NTU RGB+D 120, and Kinetics-Skeleton benchmarks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In action recognition tasks, feature diversity is essential for enhancing model generalization and performance. Existing methods typically promote feature diversity by expanding the training data in the sample space, which often leads to inefficiencies and semantic inconsistencies. To overcome these problems, we propose a novel Coarse-fine text co-guidance Diffusion model (CoCoDiff). CoCoDiff generates diverse yet semantically consistent features in the latent space by leveraging diffusion and multi-granularity textual guidance. Specifically, our approach feeds spatio-temporal features extracted from skeleton sequences into a latent diffusion model to generate diverse action representations. Meanwhile, we introduce a coarse-fine text co-guided strategy that leverages textual information from large language models (LLMs) to ensure semantic consistency between the generated features and the original inputs. It is noted that CoCoDiff operates as a plug-and-play auxiliary module during training, incurring no additional inference cost. Extensive experiments demonstrate that CoCoDiff achieves SOTA performance on skeleton-based action recognition benchmarks, including NTU RGB+D, NTU RGB+D 120 and Kinetics-Skeleton.

Problem

• Enhancing feature diversity in action recognition tasks
• Overcoming inefficiencies in sample space data expansion
• Ensuring semantic consistency in generated latent features

Innovation

• Latent diffusion model diversifies skeleton action features
• Coarse-fine text co-guidance ensures semantic consistency
• Plug-and-play module with no inference cost