MultiEYE: Dataset and Benchmark for OCT-Enhanced Retinal Disease Recognition From Fundus Images

2024-12-12

IEEE Transactions on Medical Imaging

AI Summary

Existing fundus–OCT multimodal methods rely on strictly paired data, severely limiting clinical deployability. This paper proposes a novel paradigm—“OCT-enhanced fundus image disease recognition”—which requires only unpaired fundus and OCT images during training and accepts only single-modality fundus photographs at inference. To this end, we introduce OCT-CoDA, the first unpaired cross-modal knowledge distillation framework that bridges OCT and fundus domains via interpretable semantic concepts to enable OCT-to-fundus knowledge transfer. We further establish MultiEYE, the first large-scale, multi-disease, multimodal public benchmark for fundus–OCT analysis. Extensive experiments demonstrate that our method significantly improves both classification accuracy and interpretability of fundus-based models across multiple ocular disease tasks. The code and dataset are publicly released.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Existing multi-modal learning methods on fundus and OCT images mostly require both modalities to be available and strictly paired for training and testing, which appears less practical in clinical scenarios. To expand the scope of clinical applications, we formulate a novel setting, “OCT-enhanced disease recognition from fundus images”, that allows for the use of unpaired multi-modal data during the training phase, and relies on the widespread fundus photographs for testing. To benchmark this setting, we present the first large multi-modal multi-class dataset for eye disease diagnosis, MultiEYE, and propose an OCT-assisted Conceptual Distillation Approach (OCT-CoDA), which employs semantically rich concepts to extract disease-related knowledge from OCT images and leverages them into the fundus model. Specifically, we regard the image-concept relation as a link to distill useful knowledge from OCT teacher model to fundus student model, which considerably improves the diagnostic performance based on fundus images and formulates the cross-modal knowledge transfer into an explainable process. Through extensive experiments on the multi-disease classification task, our proposed OCT-CoDA demonstrates remarkable results and interpretability, showing great potential for clinical application. Our dataset and code are available at https://github.com/xmed-lab/MultiEYE.

Problem

• Enhances retinal disease recognition using unpaired OCT and fundus images
• Introduces MultiEYE dataset for multi-modal eye disease diagnosis
• Proposes OCT-CoDA for explainable cross-modal knowledge transfer

Innovation

• Uses unpaired multi-modal data for training
• Extracts disease knowledge via OCT-CoDA
• Enhances fundus diagnosis with OCT insights