Distilling foundation models for robust and efficient models in digital pathology

2025-01-27

AI Summary

To address the high computational cost and deployment challenges of large foundation models in digital pathology, this paper proposes the first pathology-specific knowledge distillation framework, yielding a lightweight model—H0-mini. Methodologically, it integrates multi-task transfer learning, histopathological image representation distillation, and a robustness-driven PLISM evaluation mechanism. H0-mini reduces model parameters by several orders of magnitude and achieves significantly faster inference, while maintaining state-of-the-art performance on the HEST (3rd place) and EVA (5th place) benchmarks. Crucially, it demonstrates superior robustness to staining and scanner variations compared to existing SOTA models. This work establishes the first efficient distillation paradigm for pathology foundation models, enabling clinically deployable AI systems with minimal computational overhead and enhanced generalizability across real-world imaging conditions.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Other Foundations of Computer Vision
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

In recent years, the advent of foundation models (FM) for digital pathology has relied heavily on scaling the pre-training datasets and the model size, yielding large and powerful models. While it resulted in improving the performance on diverse downstream tasks, it also introduced increased computational cost and inference time. In this work, we explore the distillation of a large foundation model into a smaller one, reducing the number of parameters by several orders of magnitude. Leveraging distillation techniques, our distilled model, H0-mini, achieves nearly comparable performance to large FMs at a significantly reduced inference cost. It is evaluated on several public benchmarks, achieving 3rd place on the HEST benchmark and 5th place on the EVA benchmark. Additionally, a robustness analysis conducted on the PLISM dataset demonstrates that our distilled model reaches excellent robustness to variations in staining and scanning conditions, significantly outperforming other state-of-the art models. This opens new perspectives to design lightweight and robust models for digital pathology, without compromising on performance.

Problem

• Digital Pathology
• Resource-efficient Modeling
• Large-scale Data Processing

Innovation

• Model Distillation
• H0-mini Model
• Digital Pathology