Diagnosis of Human Object Interaction Detectors for Real World Educational Applications

2026-06-01

AI Summary

Existing human-object interaction (HOI) detectors suffer significant performance degradation in real-world educational settings due to domain-specific objects, occlusions, and complex visual conditions. This work proposes a diagnosis-driven optimization framework that introduces, for the first time, a triplet-level HOI error taxonomy and an error attribution mechanism tailored to educational scenarios. By precisely identifying model failure modes, the framework guides targeted fine-tuning of pretrained HOI models. Evaluated on the CCATT mixed-reality medical training video dataset, the approach boosts the macro-F1 score of the CDN model from 48.6 to 90.2, substantially enhancing its applicability and robustness in authentic educational environments.

Abstract

Human-object interaction (HOI) recognition is critical for automatically analyzing student behavior in complex educational environments. Although state-of-the-art (SOTA) HOI detectors perform well on benchmark datasets, their performance often degrades when deployed in real-world training environments due to domain-specific objects, occlusions, and complex visual conditions. In this paper, we introduce a diagnosis-driven framework that integrates a triplet-level HOI error taxonomy with error-factor attribution analysis for real-world educational video data. We study this problem in the context of Critical Care Air Transport Team (CCATT) mixed-reality medical training. Based on an analysis of HOI failure modes and their causes, we develop a diagnosis-informed refinement strategy for adapting pretrained HOI models to the target domain. Experiments on the CCATT dataset show that this approach improves the macro-F1 score of a pretrained CDN model from 48.6 to 90.2 through targeted refinement guided by diagnosed error factors. These results highlight the value of detailed diagnostic analysis for informing targeted adaptation of HOI models in real-world educational environments.

Problem

• Human-Object Interaction
• Real-world Educational Applications
• Domain Adaptation
• Performance Degradation
• Behavior Analysis

Innovation

• HOI diagnosis
• error taxonomy
• domain adaptation
• triplet-level analysis
• educational video understanding