Scriboora: Rethinking Human Pose Forecasting

2025-11-19

AI Summary

Human pose prediction suffers from poor reproducibility and performance degradation caused by estimation noise inherent in pose estimators. To address these challenges, we propose a unified training and evaluation framework that— for the first time—adapts modern speech modeling principles to this task. Our method employs a sequence modeling architecture integrating supervised learning with unsupervised fine-tuning, enabling end-to-end optimization directly on noisy joint coordinates output by off-the-shelf pose estimators. We introduce a novel evaluation paradigm grounded in realistic estimation noise and release corresponding noisy dataset variants to foster robustness research. Experiments demonstrate that our approach achieves significant improvements over state-of-the-art methods on standard benchmarks. Crucially, unsupervised fine-tuning is empirically validated to effectively mitigate estimator noise, substantially enhancing deployment robustness in real-world scenarios.

Technology Category

• Intelligent Robots: State Estimation
• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Biometrics, Face, Gesture & Pose

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Human pose forecasting predicts future poses based on past observations, and has many significant applications in areas such as action recognition, autonomous driving or human-robot interaction. This paper evaluates a wide range of pose forecasting algorithms in the task of absolute pose forecasting, revealing many reproducibility issues, and provides a unified training and evaluation pipeline. After drawing a high-level analogy to the task of speech understanding, it is shown that recent speech models can be efficiently adapted to the task of pose forecasting, and improve current state-of-the-art performance. At last the robustness of the models is evaluated, using noisy joint coordinates obtained from a pose estimator model, to reflect a realistic type of noise, which is more close to real-world applications. For this a new dataset variation is introduced, and it is shown that estimated poses result in a substantial performance degradation, and how much of it can be recovered again by unsupervised finetuning.

Problem

• Evaluating pose forecasting algorithms reveals reproducibility issues in human motion prediction
• Adapting speech understanding models improves state-of-the-art human pose forecasting performance
• Assessing model robustness with realistic noise from pose estimators for practical applications

Innovation

• Unified training and evaluation pipeline for pose forecasting
• Adapted speech models to improve pose forecasting performance
• Introduced dataset variation with realistic noise for robustness