Shared Semantic Codebook Distillation for Unpaired Cross-Modal Medical Classification

2026-07-29

AI Summary

This work addresses the challenge of cross-modal knowledge distillation in unpaired multimodal medical imaging, where discrepancies in patient populations and incompatible feature spaces hinder effective knowledge transfer. To overcome this, the authors propose a distillation framework based on a shared semantic codebook that maps both teacher and student modalities into a modality-agnostic discrete semantic space. Knowledge is transferred without requiring paired samples or direct feature alignment by aligning global and class-conditional distributions in this shared space. Key technical components include exponential moving average updates, entropy regularization, dead-code revival, and online codebook evolution, which jointly preserve semantic diversity while maintaining a lightweight student model. Evaluated on OCT-to-fundus photograph and CT-to-chest X-ray classification tasks, the student model achieves macro-F1 scores of 70.2 and 76.3, respectively, substantially outperforming existing approaches.

Abstract

Cross-modal knowledge distillation can transfer diagnostic knowledge from a strong but costly teacher modality to a cheaper and more deployable student modality. In medical image analysis, however, the two modalities are often unpaired: they are collected from different patient cohorts and occupy geometrically incompatible feature spaces. This makes instance-level distillation invalid and direct feature matching unreliable. To address these challenges, we propose Shared Semantic Codebook Distillation (SSCD), which compares teacher and student representations through a shared discrete codebook. Each image is represented as a distribution over a common, modality-agnostic vocabulary, and knowledge is transferred by aligning these distributions across modalities, both globally and class-conditionally, without requiring paired samples or directly comparable raw features. The codebook is evolved online by exponential moving average and kept diverse through entropy regularization and dead-code restart. At inference, all teacher-side and codebook modules are discarded, leaving only the student encoder and classifier. On two heterogeneous unpaired settings, OCT-to-fundus retinal disease classification and CT-to-chest-X-ray pneumonia classification, SSCD improves the student from 64.5 to 70.2 macro-F1 and from 73.8 to 76.3 macro-F1, respectively, outperforming all evaluated distillation baselines on both settings. Code and pretrained models are available at https://github.com/DillanImans/SSCD-unpaired-distillation

Problem

• unpaired cross-modal
• medical image classification
• knowledge distillation
• feature space incompatibility
• modality mismatch

Innovation

• Shared Semantic Codebook
• Unpaired Cross-Modal Distillation
• Modality-Agnostic Representation
• Knowledge Distillation
• Medical Image Classification