Subgroup Performance Analysis in Hidden Stratifications

2025-03-13

AI Summary

Medical AI models often exhibit implicit performance disparities across real-world patient populations, yet conventional subgroup analyses—relying on limited, predefined metadata (e.g., sex)—fail to uncover their root causes. To address this, we propose the first unsupervised implicit subgroup discovery framework for trustworthy medical AI deployment: it requires neither labels nor prior metadata, instead identifying performance-sensitive patient subgroups via feature-space clustering and quantifying diagnostic performance gaps across them. Our methodological innovations include a lightweight, scalable subgroup discovery pipeline and a novel performance disparity assessment strategy grounded in calibrated uncertainty estimation. Evaluated on chest X-ray and skin lesion classification tasks, the framework reveals inter-subgroup accuracy gaps exceeding 30%—entirely undetected by standard metadata-based analysis. This provides the first empirically validated, interpretable framework enabling fine-grained model validation and continuous monitoring in clinical AI deployment.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Humans and AI: Explainable AI (XAI) for Human Understanding
• Computer Vision: Interpretability, Explainability, and Transparency

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Machine learning (ML) models may suffer from significant performance disparities between patient groups. Identifying such disparities by monitoring performance at a granular level is crucial for safely deploying ML to each patient. Traditional subgroup analysis based on metadata can expose performance disparities only if the available metadata (e.g., patient sex) sufficiently reflects the main reasons for performance variability, which is not common. Subgroup discovery techniques that identify cohesive subgroups based on learned feature representations appear as a potential solution: They could expose hidden stratifications and provide more granular subgroup performance reports. However, subgroup discovery is challenging to evaluate even as a standalone task, as ground truth stratification labels do not exist in real data. Subgroup discovery has thus neither been applied nor evaluated for the application of subgroup performance monitoring. Here, we apply subgroup discovery for performance monitoring in chest x-ray and skin lesion classification. We propose novel evaluation strategies and show that a simplified subgroup discovery method without access to classification labels or metadata can expose larger performance disparities than traditional metadata-based subgroup analysis. We provide the first compelling evidence that subgroup discovery can serve as an important tool for comprehensive performance validation and monitoring of trustworthy AI in medicine.

Problem

• Identify performance disparities in ML models across patient groups.
• Discover hidden stratifications using learned feature representations.
• Evaluate subgroup discovery for medical AI performance monitoring.

Innovation

• Subgroup discovery for performance monitoring
• Simplified method without classification labels
• Exposes larger disparities than metadata analysis