WeCKD: Weakly-supervised Chained Distillation Network for Efficient Multimodal Medical Imaging

2025-10-16

AI Summary

Traditional knowledge distillation relies on strong teacher models and abundant labeled data, suffering from knowledge degradation and low supervision efficiency in low-resource medical imaging scenarios. To address these challenges, we propose WeCKD—the first weakly supervised chained knowledge distillation framework—comprising a cascaded model sequence that enables progressive knowledge transfer and dynamic accumulation. WeCKD introduces three key mechanisms: feature reuse, partial data sampling, and collaborative learning of multi-stage intermediate representations, substantially reducing dependency on both labeled data and powerful teachers. The method supports few-shot and weakly supervised training, demonstrating strong generalization across cross-modal medical images—including otoscopic, microscopic, and MRI modalities. Evaluated on four otolaryngological datasets, WeCKD achieves performance on par with fully supervised baselines and yields up to 23% absolute accuracy improvement over single-backbone models.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Knowledge distillation (KD) has traditionally relied on a static teacher-student framework, where a large, well-trained teacher transfers knowledge to a single student model. However, these approaches often suffer from knowledge degradation, inefficient supervision, and reliance on either a very strong teacher model or large labeled datasets, which limits their effectiveness in real-world, limited-data scenarios. To address these, we present the first-ever Weakly-supervised Chain-based KD network (WeCKD) that redefines knowledge transfer through a structured sequence of interconnected models. Unlike conventional KD, it forms a progressive distillation chain, where each model not only learns from its predecessor but also refines the knowledge before passing it forward. This structured knowledge transfer further enhances feature learning, reduces data dependency, and mitigates the limitations of one-step KD. Each model in the distillation chain is trained on only a fraction of the dataset and demonstrates that effective learning can be achieved with minimal supervision. Extensive evaluations across four otoscopic imaging datasets demonstrate that it not only matches but in many cases surpasses the performance of existing supervised methods. Experimental results on two other datasets further underscore its generalization across diverse medical imaging modalities, including microscopic and magnetic resonance imaging. Furthermore, our evaluations resulted in cumulative accuracy gains of up to +23% over a single backbone trained on the same limited data, which highlights its potential for real-world adoption.

Problem

• Overcoming knowledge degradation in traditional knowledge distillation methods
• Reducing dependency on large labeled datasets for medical imaging
• Enhancing multimodal medical image analysis with limited supervision

Innovation

• Weakly-supervised chain-based distillation for efficient learning
• Progressive interconnected models refine knowledge sequentially
• Reduces data dependency while enhancing feature learning