FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs

2026-09-27

AI Summary

This study addresses the challenge of evaluating the reliability of retinal AI models under distribution shifts, varying clinical definitions, and diverse patient subgroups. By integrating ten public fundus datasets into a cross-dataset benchmark, it presents the first systematic comparison of three architectural paradigms: vision encoders, vision-language models (VLMs), and multimodal large language models (MLLMs). Models are adapted via LoRA supervised fine-tuning and evaluated across ranking performance, calibration, subgroup fairness, and image quality robustness. Results indicate that no single model family universally dominates, with general-purpose vision encoders achieving the best overall performance. Although LoRA fine-tuning improves task-specific metrics, it induces heterogeneous degradation in calibration on external data, revealing a fundamental trade-off between generalization and reliability.

Abstract

Progress in AI-based retinal image analysis has advanced with foundation models, yet evaluating their reliability remains challenging. Performance reported on a single dataset does not capture how models behave under dataset shift, across clinical definitions, or for different patient subgroups. This limitation is particularly critical in medical imaging analysis, where robustness, calibration, and fairness are essential for safe deployment. We introduce FOCUS (Foundation Ophthalmic Cross-Dataset Understanding under Shift), a cross-dataset benchmark for evaluating retinal fundus models that considers vision-only encoder models (VM), vision-language dual-encoder models (VLM), and multimodal large language models (MLLM). FOCUS harmonizes binary diabetic retinopathy, referable diabetic retinopathy, and glaucomatous optic neuropathy tasks across ten public datasets spanning diverse geographies, acquisition conditions, and label protocols. The benchmark evaluates models through a unified analysis layer that measures ranking performance, calibration, subgroup disparities, and image-quality robustness. We present a large-scale evaluation covering 532 base configurations and 228 MLLM configurations adapted through supervised fine-tuning with low-rank adaptation (LoRA). Results show that no model family consistently dominates across tasks and datasets: general VM encoders achieve the strongest average ranking performance, medical MLLMs are competitive but variable, and dual encoder VLMs benefit substantially from lightweight adaptation. Fine-tuning improves in-domain performance but exhibits heterogeneous transfer to external datasets, particularly in calibration. These findings demonstrate that retinal model evaluation is inherently multidimensional. FOCUS provides a practical framework and public benchmark to assess generalization, reliability, and robustness beyond single-dataset leaderboards

Problem

• retinal image analysis
• cross-dataset generalization
• model reliability
• robustness evaluation
• multimodal LLMs

Innovation

• Cross-dataset Benchmark
• Multimodal Large Language Models
• Low-Rank Adaptation
• Model Calibration
• Retinal Image Analysis