Causal-Inspired Multitask Learning for Video-Based Human Pose Estimation

2025-01-24

AI Summary

Video human pose estimation suffers from inadequate causal modeling, leading to limited robustness and interpretability in complex scenarios such as occlusion and motion blur. Method: This paper pioneers the integration of causal inference into this task, proposing a two-stage causal-inspired multi-task learning framework. In Stage I, self-supervised auxiliary tasks endow the model with causal reasoning capability. In Stage II, a Token Causal Importance Selection module and a Non-Causal Token Clustering module explicitly separate causal from non-causal features. Contribution/Results: The framework enables causal-driven joint optimization across multiple tasks, achieving state-of-the-art performance on three major benchmarks. Notably, it significantly improves estimation robustness under challenging conditions—including severe occlusion and motion blur—while providing interpretable attention mechanisms grounded in causal feature attribution.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Computer Vision: Visual Reasoning & Symbolic Representations

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Video-based human pose estimation has long been a fundamental yet challenging problem in computer vision. Previous studies focus on spatio-temporal modeling through the enhancement of architecture design and optimization strategies. However, they overlook the causal relationships in the joints, leading to models that may be overly tailored and thus estimate poorly to challenging scenes. Therefore, adequate causal reasoning capability, coupled with good interpretability of model, are both indispensable and prerequisite for achieving reliable results. In this paper, we pioneer a causal perspective on pose estimation and introduce a causal-inspired multitask learning framework, consisting of two stages. extit{In the first stage}, we try to endow the model with causal spatio-temporal modeling ability by introducing two self-supervision auxiliary tasks. Specifically, these auxiliary tasks enable the network to infer challenging keypoints based on observed keypoint information, thereby imbuing causal reasoning capabilities into the model and making it robust to challenging scenes. extit{In the second stage}, we argue that not all feature tokens contribute equally to pose estimation. Prioritizing causal (keypoint-relevant) tokens is crucial to achieve reliable results, which could improve the interpretability of the model. To this end, we propose a Token Causal Importance Selection module to identify the causal tokens and non-causal tokens ( extit{e.g.}, background and objects). Additionally, non-causal tokens could provide potentially beneficial cues but may be redundant. We further introduce a non-causal tokens clustering module to merge the similar non-causal tokens. Extensive experiments show that our method outperforms state-of-the-art methods on three large-scale benchmark datasets.

Problem

• Human Pose Estimation
• Complex Scenes
• Body Part Interrelations

Innovation

• Causality-inspired Multitask Learning
• Body Part Interrelations
• Information Prioritization and Aggregation