Evaluating the evaluators: Towards human-aligned metrics for missing markers reconstruction

2024-10-18

arXiv.org

AI Summary

Existing evaluation metrics—primarily mean squared error (MSE)—for marker reconstruction in optical motion capture exhibit severe misalignment with human perceptual quality. Method: This paper first systematically identifies the fundamental perceptual misalignment of conventional metrics; then, grounded in perceptual psychology principles, it designs and validates a novel set of perception-aware evaluation metrics; finally, it establishes a rigorous comparative evaluation framework across multiple models and datasets via large-scale human subjective rating experiments, analyzed using Spearman rank correlation and Pearson linear correlation coefficient (PLCC). Contribution/Results: The proposed metrics achieve significantly improved consistency with human judgment, yielding an average PLCC improvement of 0.42. This work establishes the first reliable, perceptually aligned benchmark for motion capture reconstruction evaluation.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Evaluation and Analysis
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Animation data is often obtained through optical motion capture systems, which utilize a multitude of cameras to establish the position of optical markers. However, system errors or occlusions can result in missing markers, the manual cleaning of which can be time-consuming. This has sparked interest in machine learning-based solutions for missing marker reconstruction in the academic community. Most academic papers utilize a simplistic mean square error as the main metric. In this paper, we show that this metric does not correlate with subjective perception of the fill quality. Additionally, we introduce and evaluate a set of better-correlated metrics that can drive progress in the field.

Problem

• Develop human-aligned metrics for missing marker reconstruction
• Address limitations of mean square error in animation data
• Evaluate better-correlated metrics for fill quality perception

Innovation

• Machine learning for missing marker reconstruction
• New metrics for subjective fill quality
• Evaluation beyond mean square error