ReMoE: Report-Guided Mixture-of-Experts for Multimodal OCT/OCTA Anomaly Detection

2026-07-31

AI Summary

This work addresses a critical limitation in existing unsupervised multimodal OCT/OCTA anomaly detection methods, which rely solely on visual features or reconstruction residuals and overlook the rich semantic structure embedded in normal clinical reports, thereby constraining anomaly scores to superficial deviations. To bridge this gap, we propose the first report-guided mixture-of-experts framework that integrates semantic information from medical reports into anomaly detection. Specifically, we employ knowledge distillation to transfer semantic priors from normal reports to an image-to-text student model, establishing modality-aware priors. Furthermore, we introduce a Report-Guided Modality Modulation (RMM) mechanism that dynamically adjusts multimodal feature representations through expert routing conditioned on report semantics. Our approach achieves state-of-the-art performance on both a private paired OCT/OCTA dataset and the public OCTA500-3MM benchmark.

Abstract

Multimodal medical anomaly detection identifies samples deviating from normal patterns, where scarce abnormal cases make normality modeling from normal data practical. In retinal Optical Coherence Tomography (OCT) and OCT Angiography (OCTA) anomaly detection, existing unsupervised methods rely on visual feature distributions, reconstruction residuals, or encoder-decoder discrepancies, making anomaly scores depend on appearance-level deviations, while multimodal normality also contains semantic organization described in normal medical reports. To this end, we propose Report-Guided Mixture-of-Experts (ReMoE), which distills normal report semantics into an image-to-text prior student, builds modality-aware priors, and uses Report-Guided Modality Modulation (RMM) to modulate features through mixture-of-experts routing. Experiments on a private OCT/OCTA dataset with paired normal reports and a public OCTA500-3MM setting using a fixed normal report demonstrate state-of-the-art performance.

Problem

• multimodal anomaly detection
• OCT/OCTA
• medical report semantics
• unsupervised learning
• normality modeling

Innovation

• Mixture-of-Experts
• Report-Guided Modality Modulation
• Multimodal Anomaly Detection
• Medical Report Semantics
• OCT/OCTA