Metamorphic Testing for Clinical ML Models: A Framework Proposal and Pilot Study

2026-07-24

AI Summary

This study addresses the critical gap that clinical machine learning models, despite achieving high AUROC scores (0.849–0.900), often produce predictions inconsistent with medical common sense and lack methods for verifying the reasonableness of individual predictions without ground-truth labels. The authors propose the first systematic framework of 12 metamorphic relations (MRs) derived from authoritative clinical guidelines for ICU prediction tasks, accompanied by a five-tier validation strategy to ensure their clinical validity. Evaluations on MIMIC-III/IV and UCI Heart Disease datasets, using metamorphic testing and fault injection, reveal that MT violation rates range from 27% to 87%. Notably, injecting sign errors into blood pressure features leaves AUROC nearly unchanged but increases MT violation rates by 31–67 percentage points, effectively exposing behavioral flaws invisible to conventional performance metrics.

Abstract

Machine learning models for clinical prediction tasks, such as in-hospital mortality and sepsis onset, routinely achieve high AUROC scores. However, AUROC measures ranking performance rather than clinical sensibility. A model may rank patients correctly overall while predicting a lower mortality risk when a patient's SOFA score worsens, contradicting established medical knowledge. This paper proposes applying metamorphic testing (MT) to clinical machine learning models to evaluate behavioral correctness without requiring ground-truth labels for individual predictions. We design a catalog of 12 candidate metamorphic relations (MRs) for three ICU prediction tasks using the MIMIC-III and MIMIC-IV datasets, with each MR grounded in an authoritative clinical guideline. We further propose a five-layer validation strategy to ensure that MRs are clinically sound before deployment. As a feasibility study, we evaluate the approach on the UCI Heart Disease dataset. Although the three clinical models achieve strong predictive performance (AUROC = 0.849-0.900), they exhibit MT violation rates ranging from 27% to 87% across five pilot MRs. An injected-fault experiment further shows that a sign-negation error in a blood pressure feature remains undetected by AUROC but increases the MT violation rate by 31-67 percentage points. These findings suggest that metamorphic testing provides a valuable complement to conventional performance metrics for assessing the behavioral correctness of clinical prediction models.

Problem

• metamorphic testing
• clinical machine learning
• behavioral correctness
• prediction reliability
• medical knowledge consistency

Innovation

• Metamorphic Testing
• Clinical Machine Learning
• Metamorphic Relations
• Behavioral Correctness
• Model Validation