PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Image

2026-07-21

AI Summary

Existing pathology benchmarks predominantly rely on pre-cropped image patches or pre-extracted features, limiting their ability to evaluate models’ capacity to actively acquire diagnostic evidence from gigapixel whole-slide images (WSIs). This work proposes PathAgentBench, the first WSI-based benchmark for assessing active evidence-seeking capabilities, structured around pathologist-annotated diagnostic trees that hierarchically organize multi-scale regions, findings, and diagnoses. The framework systematically evaluates vision-language models across four core competencies: evidence interpretation, verification, localization, and integration. Incorporating multi-scale nested regions, cross-modal image–text alignment, text-guided localization, and autonomous sliding-window exploration, PathAgentBench reveals that while state-of-the-art open-source models achieve over 93% accuracy in multi-scale reasoning on TCGA and private breast cancer WSIs, their region localization performance remains poor (mIoU < 0.09), and autonomous exploration success sharply declines with increasing magnification—highlighting evidence acquisition as a critical bottleneck.

Abstract

Whole-slide image (WSI) diagnosis requires identifying diagnostically relevant regions, examining them across magnifications, and integrating multi-scale evidence. However, most existing pathology benchmarks evaluate models on pre-cropped patches or pre-extracted slide features, leaving their ability to acquire evidence directly from gigapixel WSIs largely untested. We introduce PathAgentBench, a benchmark for evaluating evidence-seeking vision-language models (VLMs) across four complementary capabilities: image-to-text matching for evidence interpretation, text-to-image retrieval for evidence verification, diagnostic-region localization for evidence acquisition, and multi-scale reasoning for evidence integration. The benchmark is organized as a diagnostic tree that links nested regions across magnifications with scale-specific findings and path-level diagnoses. It contains 1,822 TCGA WSIs and 17,135 diagnostic paths annotated by ten board-certified pathologists. An additional private cohort of 190 breast cancer WSIs with detailed annotations is used to evaluate autonomous whole-slide exploration. We evaluate 20 general-purpose, medical, and pathology-specialized models. Leading open-weight models achieve over 93% accuracy in multi-scale reasoning and over 50% accuracy in both cross-modal matching tasks. In contrast, diagnostic-region localization remains challenging: the best text-guided mean intersection-over-union is below 0.09, underperforming a simple center-based heuristic. During autonomous exploration, the unconditional hit rate decreases from 0.522 at low magnification to 0.185 at intermediate magnification and 0.020 at high magnification. These results reveal a pronounced gap between reasoning over curated evidence and acquiring that evidence directly from WSIs. PathAgentBench provides a unified framework for measuring and improving evidence-seeking pathology models.

Problem

• whole-slide image
• evidence-seeking
• vision-language models
• diagnostic-region localization
• multi-scale reasoning

Innovation

• evidence-seeking
• whole-slide image
• vision-language model
• multi-scale reasoning
• diagnostic tree