Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results

2025-05-13

AI Summary

Clinical deployment of medical image segmentation is hindered by annotator variability, leading to subjective ground truths, miscalibrated models, and unreliable uncertainty estimates. Method: We propose the first multi-annotator evaluation framework integrating both consensus and disagreement ground truths. Our approach introduces a novel calibration–uncertainty co-evaluation paradigm jointly leveraging Expected Calibration Error (ECE) and Continuous Ranked Probability Score (CRPS), and employs pre-trained transfer learning with robust multi-source data training to enhance model reliability. Contribution/Results: We systematically demonstrate a strong correlation between calibration quality and segmentation performance (Dice score). The method achieves state-of-the-art (SOTA) segmentation accuracy while significantly improving calibration and confidence–accuracy alignment. Empirical validation confirms that multi-annotator strategies are essential for building clinically trustworthy segmentation models.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Segmentation
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Deep learning (DL) has become the dominant approach for medical image segmentation, yet ensuring the reliability and clinical applicability of these models requires addressing key challenges such as annotation variability, calibration, and uncertainty estimation. This is why we created the Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS), which highlights the critical role of multiple annotators in establishing a more comprehensive ground truth, emphasizing that segmentation is inherently subjective and that leveraging inter-annotator variability is essential for robust model evaluation. Seven teams participated in the challenge, submitting a variety of DL models evaluated using metrics such as Dice Similarity Coefficient (DSC), Expected Calibration Error (ECE), and Continuous Ranked Probability Score (CRPS). By incorporating consensus and dissensus ground truth, we assess how DL models handle uncertainty and whether their confidence estimates align with true segmentation performance. Our findings reinforce the importance of well-calibrated models, as better calibration is strongly correlated with the quality of the results. Furthermore, we demonstrate that segmentation models trained on diverse datasets and enriched with pre-trained knowledge exhibit greater robustness, particularly in cases deviating from standard anatomical structures. Notably, the best-performing models achieved high DSC and well-calibrated uncertainty estimates. This work underscores the need for multi-annotator ground truth, thorough calibration assessments, and uncertainty-aware evaluations to develop trustworthy and clinically reliable DL-based medical image segmentation models.

Problem

• Addressing annotation variability in medical image segmentation
• Ensuring model calibration and uncertainty estimation reliability
• Leveraging multi-annotator data for robust segmentation evaluation

Innovation

• Leveraging multi-annotator variability for robust evaluation
• Using diverse datasets and pre-trained knowledge for robustness
• Incorporating calibration and uncertainty metrics for reliability