UltraBench 2: Towards Robust Evaluation of Vision Foundation Models on Ultrasound

2026-09-23

AI Summary

This study addresses the lagging, fragmented, and inconsistent evaluation of ultrasound foundation models by establishing the first highly comprehensive and reproducible unified benchmark. The proposed framework encompasses diverse anatomical structures and tasks, introducing standardized testing protocols for classification and segmentation to systematically compare existing vision foundation models. Experimental results demonstrate that ultrasound-specific pretrained models maintain a performance advantage in classification, whereas general-purpose state-of-the-art models have achieved parity in segmentation tasks. By filling the critical gap in ultrasound benchmarking, this work provides a reliable evaluation paradigm to guide future model selection and development in the field.

Abstract

Benchmarking is an increasingly critical part of research in machine learning and the domains where it is applied, including healthcare. Yet, despite the steady development of new ultrasound foundation models in recent years, the development of well-designed benchmarks to evaluate them has lagged behind. This deficiency has led to fragmented and inconsistent evaluations of competing models, making it difficult to measure progress. To address this issue, we introduce UltraBench 2, a comprehensive benchmark with wide anatomical and task coverage, and a focus on standardization, reproducibility, and ease-of-use. Using this benchmark, we compare existing vision foundation models for ultrasound image analysis. Our analyses demonstrate that ultrasound-specific pretraining still leads on classification, but that state-of-the-art general-purpose models have drawn level on segmentation.

Problem

• Ultrasound Foundation Models
• Benchmarking
• Evaluation
• Vision Foundation Models

Innovation

• Ultrasound Foundation Models
• Benchmarking
• Vision Foundation Models
• Standardization
• Reproducibility