Representation Transfer of Foundation Models for Ultra-Widefield Retinal Imaging

2026-08-01

AI Summary

This study systematically evaluates the impact of different pretraining strategies on the transferability of Vision Transformers (ViTs) for weakly supervised disease classification in ultra-widefield fundus images. Within a unified patch-level multiple instance learning framework, the authors compare ViT-B models pretrained on ImageNet-1k using supervised learning, Masked Autoencoders (MAE), and self-distillation, and further incorporate the larger-scale DINOv3 model. The results demonstrate that the choice of pretraining strategy not only significantly affects the transfer performance of frozen features but also alters the aggregation behavior of patch-level evidence. DINOv3 achieves the best performance across five-class diabetic retinopathy grading tasks, yielding a quadratic weighted Kappa of 0.863, while partially fine-tuned MAE effectively narrows the performance gap with other approaches.

Abstract

Despite the widespread adoption of foundation models as feature extractors for medical imaging, relatively little is understood about how different pretraining strategies influence the transferability of learned representations to weakly supervised ophthalmic imaging tasks. We investigate this question in ultra-widefield (UWF) retinal imaging by evaluating foundation model representations within a patch-based multiple instance learning (MIL) framework for disease classification on UWF images. We compare Vision Transformer encoders pretrained with supervised, Masked Autoencoder (MAE), and self-distillation objectives, while keeping the downstream aggregation architecture unchanged. Within a controlled comparison of ViT-B encoders pretrained on ImageNet-1k, the choice of pretraining objective substantially influenced frozen representation transfer, with supervised and self-distillation-based models outperforming MAE. A contemporary DINOv3 model pretrained at a larger scale achieved the strongest overall performance, with a quadratic weighted kappa of 0.863 for five-class diabetic retinopathy grading, comparable with DINOv1. Attention analysis further revealed distinct patch-aggregation behaviours associated with the different pretrained representations, while partial fine-tuning substantially reduced the performance gap for MAE. These findings suggest that pretraining strategy influences both representation transferability and the subsequent aggregation of patch-level evidence within MIL, resulting in differences in downstream classification performance.

Problem

• foundation models
• ultra-widefield retinal imaging
• representation transfer
• pretraining strategies
• weakly supervised learning

Innovation

• foundation models
• pretraining strategy
• ultra-widefield retinal imaging
• multiple instance learning
• representation transfer