OrthKD: Extracting Generalized Clinical Knowledge from Heterogeneous Teachers for Lightweight Deployment

2026-07-28

AI Summary

This work addresses the dual challenges of model compactness and robustness under distribution shift in diabetic retinopathy screening for primary care settings. The authors propose OrthKD, a selective knowledge distillation framework that fuses a strong CNN teacher and a weak ViT teacher: the strong teacher undergoes full supervision-based distillation, while the weak teacher contributes only at the feature level. To enhance complementary knowledge transfer, an orthogonality constraint is imposed on teacher-specific projection heads. The resulting MobileNetV3-based student model (5.4M parameters) achieves a quadratic weighted kappa (QWK) of 0.885 on EyePACS and demonstrates remarkable zero-shot generalization to Messidor-2, improving QWK from 0.507 to 0.728, while maintaining high referral AUC and well-calibrated predictions—highlighting its balanced advantages in accuracy, safety, and cross-domain robustness.

Abstract

Deploying diabetic retinopathy (DR) screening models in primary care requires edge-efficient systems that remain accurate, safe, and reliable under domain shift. Multi-teacher knowledge distillation (KD) is a natural compression strategy, but existing approaches largely assume that all teachers provide equally trustworthy supervision. In our setting, this assumption fails: a strong CNN teacher (EfficientNet-B3, 0.876 QWK) and a weaker Transformer teacher (Swin-Base, 0.830 QWK) are complementary, yet the Transformer's logits can still mislead the student. We therefore propose OrthKD, a selective-trust distillation framework that transfers full supervision from the strong CNN, uses feature-only distillation from the weak ViT, and enforces orthogonality between teacher-specific student projections to encourage complementary rather than redundant evidence. This design preserves local lesion precision, injects global structural context, and improves robustness to distribution shift. On 132,049 retinal images, a 5.4M-parameter MobileNetV3 student reaches 0.885 QWK on EyePACS and improves zero-shot Messidor-2 performance from 0.507 to 0.728 QWK, while also achieving strong referral AUC and calibration. These results show that selectively distilling heterogeneous teachers can enable practical DR screening on resource-constrained devices.

Problem

• diabetic retinopathy screening
• knowledge distillation
• domain shift
• heterogeneous teachers
• lightweight deployment

Innovation

• Orthogonal Knowledge Distillation
• Heterogeneous Teachers
• Selective Trust
• Domain Shift Robustness
• Lightweight Deployment