Knowledge-guided Disentanglement with Atomic Actions for Action Recognition

2026-07-27

AI Summary

This work addresses the challenges of recognizing multiple concurrent fine-grained actions in complex scenes, where existing methods struggle to model the internal structure of actions and their subtle semantic interactions. To this end, we propose a large language model (LLM)-driven framework that explicitly decomposes action labels into atomic actions endowed with spatiotemporal semantics. We introduce a Knowledge Injection Module (KIM) to incorporate external semantic knowledge and a Knowledge Disentanglement Module (KDM) coupled with a knowledge distillation (KD) loss to achieve precise disentanglement of action representations. The proposed approach significantly enhances feature discriminability and interpretability, achieving state-of-the-art performance on multilabel action recognition benchmarks. Moreover, the designed modules demonstrate strong generalizability and can be flexibly integrated into existing architectures.

Abstract

Action recognition in complex scenes often involves multiple concurrent fine-grained actions, making it challenging to model internal action structures. Most existing methods rely on holistic representations, which are insufficient for capturing subtle interactions and fine-grained semantics. While recent prompt-based approaches introduce disentanglement, they lack explicit semantic guidance, and methods based solely on visual or structured cues remain coarse-grained. In this paper, we propose Knowledge-guided Disentanglement with Atomic Actions (KDA), which leverages fine-grained semantic knowledge to enhance action representations and enable more precise disentanglement. Specifically, we use Large Language Models (LLMs) to decompose action labels into atomic actions, providing explicit spatial-temporal semantics. A Knowledge Injection Module (KIM) first integrates atomic action knowledge into video features. Based on this enhanced representation, a Knowledge Disentanglement Module (KDM) further disentangles atomic action knowledge to produce more precise semantic guidance for action disentanglement. A Knowledge Disentanglement Loss (KD Loss) is introduced to encourage clearer disentanglement of knowledge components within KDM. Extensive experiments demonstrate that KDA improves feature discriminability and achieves state-of-the-art performance on multi-label action recognition benchmarks. Moreover, KIM and KDM can be readily integrated into other methods, demonstrating strong generality.

Problem

• action recognition
• fine-grained actions
• disentanglement
• semantic guidance
• complex scenes

Innovation

• atomic actions
• knowledge-guided disentanglement
• large language models
• action recognition
• semantic decomposition