Quality Action Assurance: Multimodal Verification of Examiner Claims in VR OSCEs

2026-07-21

AI Summary

This study addresses the subjectivity, fatigue, and cognitive biases inherent in human raters during Virtual Reality Objective Structured Clinical Examinations (VR OSCEs), as well as the lack of interpretability in traditional validation approaches. To this end, it introduces the first automated fact-checking framework tailored to OSCE examiner statements. The proposed method integrates multimodal data—comprising video recordings, VR interaction logs, and standardized patient (actor) inputs—to construct a ground-truth sequence of clinical events. A constrained temporal action alignment model enables precise action localization and actor attribution, while a large language model extracts examiner claims and performs factual verification against the established ground truth. Experimental results demonstrate that the system achieves 99.2% ± 0.7% Actor F1 and 93.4% ± 1.9% W@16 under five-fold cross-validation, detects erroneous examiner judgments with 70.0% precision and 76.7% recall, and significantly improves factual accuracy from 39.2% to 79.2%, thereby substantially enhancing both objectivity and interpretability in OSCE scoring.

Abstract

Objective Structured Clinical Examinations (OSCEs) are the gold standard for assessing clinical competence, yet scoring remains vulnerable to examiner subjectivity, fatigue, and cognitive bias. Standard examiner validation via inter-rater statistics lacks explanatory power regarding the source of errors, as it neither analyzes examiner reasoning nor verifies examiner claims against actual events. Thus, we introduce Quality Action Assurance (QAA), a multimodal framework that verifies examiner claims in Virtual Reality (VR) pediatric OSCEs by comparing actions claimed by examiners against the true sequence of events, constructed from video, VR logs, and actor data. QAA combines a constrained temporal action alignment model, which performs action localization and actor source attribution, with a large language model that extracts examiner claims and checks them against the record. Across a 5-fold cross-validation, QAA achieves 99.2% $\pm$ 0.7% Actor F1 and 93.4% $\pm$ 1.9% W@16 for temporal alignment. Overall, QAA detects examiner errors with 70.0% precision and 76.7% recall, improving factual correctness from 39.2% to 79.2%, enabling fairer OSCE assessment.

Problem

• OSCE
• examiner bias
• subjectivity
• clinical assessment
• scoring validation

Innovation

• multimodal verification
• temporal action alignment
• large language model
• VR OSCE
• examiner claim validation